d1.2 criteria to assess how patient engagement ... · d1.2 criteria to assess how patient...

D1.2 Criteria to assess how patient engagement expectations are met

777450 - PARADIGM

Patients Active in Research and Dialogues for an Improved Generation of Medicines

WP1 – Defining stakeholders’ preferences, needs and expectations Task 1.2 Co-Prioritize minimal expectations for patient engagement across key stakeholders

Task 1.3: Develop a set of criteria to assess patient engagement practices and processes against minimal expectations

Lead contributor Maria Jose Vicente Edo (IACS) [email protected]

Lucia Prieto Remón (IACS)

On Behalf of the IACS Team

Other contributors Chi Pakarinen (The Synergist) (WP1 co-leads)

Alexandra Moutet (UCB) (WP1 co-leads)

Ana Diaz (AE)

Begoña Nafria (SJD)

Elisa Ferrer (EURORDIS)

Neil Bertelsen (HTA)

2

Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y

Experts Panel for the Delphi on Patient Engagement in Setting Research Priorities

NAME SURNAME INSTITUTIONLara Bloom EUPATI

Helen Rochford-Brenan AlzheimerEurope

Violeta Stoyanova MedicinesEvaluationBoard(Netherlands)

Matt Bradley GSK

Malca Resnick Pfizer

Emmanuelle Logette AssociationScléroseTubéreusedeBourneville(ASTB)

Blanca Novella ComunidaddeMadrid

Pedro AbizandaSoler AlbaceteUniversityHospital

Ana Calpe FundacionSanJoandeDeu

Encarnación González FundacionSanJoandeDeu

Daniel O'Connor Medicines&HealthcareproductsRegulatoryAgency(UK)

Heidrun HILDEBRAND Bayer

Paola Mosconi IstitutodiRicercheFarmacologicheMarioNegriIRCCS

Camilla KroghLauritzen LEOPharma

Kevin McCarthy European Commission-EuropeAid Development &Cooperation(DEVCO)

Tomáš Doležal ValueOutcomesCompany

Joseph Collins Sanofi

Daria Julkowska AgenceNationaleRecherche/E-Rare

Experts panel for the Delphi on Patient Engagement in the Designing of Clinical trials

NAME SURNAME INSTITUTION

Birgit Bauer EUPATI

Maria Piggin EUPATI

Natasha Ratcliffe Parkinson'sUK

Heidi Muller Johnson&Johnson

Gautam Gupta Pfizer

FrankMichael Amort EATG

3


Tatiana Massarrah GregorioMarañónHospital

Stefanie Breitenstein Bayer

Victoria Thomas NICE

Rob Camp EURORDIS

Ana RuizCasado PuertadeHierroUniversityHospital

Nieves Tejeda FundacionSanJoandeDeu

Sofia Junquera FundacionSanJoandeDeu

Pamela Tenaerts ClinicalTrialsTransformationInitiative

Caridad Pontes CatSalut

Fabian Somers UCB

Jeannett Dimsits NovoNordisk

Juan Erviti ServicioNavarrodeSalud

Cynthia Grossman FasterCures

Sharareh Hosseinzadeh Novartis

Emilie Voltz Novartis

Annette Christoph Grunenthal

Debra Michaels DIA

Inge Schalkers Hartenraad

Traceann Rose Children'sTumorFoundation(CTF)

Experts panel for the Delphi on Patient Engagement in Early Dialogues with HTA and Regulators

NAME SURNAME INSTITUTION

Chris Roberts AlzheimerEurope

Jayne Goodrick AlzheimerEurope

Jenny Wilson GSK

Alicia Granados Sanofi

Adam Heatfield Pfizer

Marie-Pierre Gagnon UniversitéLaval

François Houÿez EURORDIS

Dimitrios Athanasiou EURORDIS/WorldDuchenneOrganization/UPPMD.

Encarnación CruzMartos ComunidaddeMadrid

Claire Champeix Eurocarers

Beatrix Friedeberg AmgenB.V.

Sandra Nestler-Parr ISPOR

4


Deborah Morrison NICE

Nathalie Bere EMA

Luciana Ballini RegioneEmilia-Romagna

Michelle Mujoomdar CADTH

Isabelle Huys KatholiekeUniversiteitLeuven

Nigel Cook NovartisPharma&IMIPREFER

Due date 30/05/2019 Delivery date 20/05/2019 Deliverable type R Dissemination level PU

Description of Action Version Date V1.1 XXX

5


Table of Contents

1. Publishable Summary.........................................................................................................................................8

2. Introduction............................................................................................................................................................10

3. Methodology..........................................................................................................................................................10

4. Results.....................................................................................................................................................................14

4.1. Representativeness of the three Delphi Expert Panels...........................................................14

4.2. Delphi on Patient Engagement in Setting Research Priorities............................................15

4.3. Delphi on Patient Engagement in Designing Clinical trials....................................................17

4.4. Delphi on Patient Engagement in Early Dialogues with HTA & Regulators................18

5. Discussion..............................................................................................................................................................20

6. Proposal for Utilising the Criteria..............................................................................................................22

Annex 1. List of sources...............................................................................................................................................24

Annex 2. Delphi questionnaires...............................................................................................................................30

Annex 3. Panel stakeholder profile........................................................................................................................42

Annex 4. Glossary for the Delphi in PE in Medicine R&D lifecycle.....................................................45

Annex 5. Results of the Delphi on PE in Setting Research Priorities................................................51

Annex 6. Results of the Delphi on PE in Designing Clinical Trials......................................................65

Annex 7. Results of the Delphi on PE in Early Dialogues........................................................................79

6


Document History

Version Date Description V0.1 01/04/2019 First Draft V0.2 08/04/2019 Comments from experts panels V0.3 06/04/2019 Comments from first round reviewers V0.4 20/05/2019 Draft edited based on comments V1.0 30/05/2019 Final Version

7


Partners of the PARADIGM Consortium are referred to herein according to the following codes: - EPF. EUROPEAN PATIENTS FORUM (Luxembourg) – Project Coordinator - EURORDIS. EUROPEAN ORGANISATION FOR RARE DISEASES ASSOCIATION (France) - EATG. EUROPEAN AIDS TREATMENT GROUP (Germany) - AE. ALZHEIMER EUROPE (Luxembourg) - AIFA. AGENZIA ITALIANA DEL FARMACO (Italy) - HTAi. HEALTH TECHNOLOGY ASSESSMENT INTERNATIONAL (Canada) - IACS. INSTITUTO ARAGONES DE CIENCIAS DE LA SALUD (Spain) - FSJD. FUNDACIO SANT JOAN DE DEU (Spain) - VU-ATHENA. STICHTING VU (The Netherlands) - UOXF-CASMI. THE CHANCELLOR, MASTERS AND SCHOLARS OF THE UNIVERSITY OF

OXFORD (United Kingdom) - EFGCP. EUROPEAN FORUM FOR GOOD CLINICAL PRACTICE (Belgium) - SYNERGIST. THE SYNERGIST (Belgium) - SYNAPSE. SYNAPSE RESEARCH MANAGEMENT PARTNERS SL (Spain) - EFPIA. EUROPEAN FEDERATION OF PHARMACEUTICAL INDUSTRIES AND

ASSOCIATIONS (Belgium) - Project Leader - MSD Corp. MERCK SHARP & DOHME CORP (United States) - UCB. UCB BIOPHARMA SPRL (Belgium) - ABPI. THE ASSOCIATION OF THE BRITISH PHARMACEUTICAL INDUSTRY (United Kingdom) - AMGEN. AMGEN LIMITED (United Kingdom) - BAYER. BAYER AKTIENGESELLSCHAFT (Germany) - GSK. GLAXOSMITHKLINE RESEARCH AND DEVELOPMENT (United Kingdom) - GRT. GRUENENTHAL GMBH (Germany) - JANSSEN. JANSSEN PHARMACEUTICA NV (Belgium) - LILLY. Eli Lilly and Company Limited (United Kingdom) - LUNDBECK. H. LUNDBECK AS (Denmark) - MERCK. MERCK KOMMANDITGESELLSCHAFT AUF AKTIEN (Germany) - NOVO NORDISK. NOVO NORDISK A/S (Denmark) - PFIZER. PFIZER LIMITED (United Kingdom) - ROCHE. F. HOFFMANN-LA ROCHE AG (Switzerland) - SERVIER. INSTITUT DE RECHERCHES INTERNATIONALES SERVIER (France) - VFA. VERBAND FORSCHENDER ARZNEIMITTELHERSTELLER EV (Germany) - SARD. SANOFI-AVENTIS RECHERCHE & DEVELOPPEMENT (France) - NOVARTIS. NOVARTIS PHARMA AG (Switzerland) - COVANCE. COVANCE LABORATORIES LTD (United Kingdom) - ALEXION. ALEXION SERVICES EUROPE (Belgium)

§ Consortium. The PARADIGM Consortium, comprising the above-mentioned legal entities § Consortium Agreement. Agreement concluded amongst PARADIGM participants for the implementation

of the Grant Agreement. Such an agreement shall not affect the parties’ obligations to the Community and/or to one another arising from the Grant Agreement.

8


1. Publishable Summary

ThisreportdescribestheworkthatwasdoneaspartofPARADIGM’s1workpackage1(WP1),which aimed to identify and define stakeholders’ preferences, needs and expectations inpatient engagement (PE) through an online survey and focus group consultations thatwereconductedin20182.

The objectives of thisworkwere two-fold: firstly to prioritise those needs and expectationsidentified,andsecondly,withthehelpoftopicexperts,turnthoseprioritisedneedsintoasetofcriteriaforagoodPEactivity.

TheDelphimethodology3wasused as themain tool to reach these objectives. Three expertpanelgroupswereformedtomatchthethreedecision-makingpointsParadigmprojectfocuseson[settingresearchpriorities(SRP),thedesignofclinicaltrials(DCT)andearlydialogueswithHealthTechnologyAssessment(HTA)bodiesandregulators(ED)]4.

Each expert group went through two online survey rounds, and a subsequent meeting inBrusselswhereexperts(whocompletedbothonlinerounds)discussedtoreachconsensusonthe final criteria toassessPEpractices.The format,procedureandmethodologyof all threemeetingswereidentical.

Each group produced a set of weighted criteria and categories for their dedicated decision-makingpoint.TheweightsallocatedrepresenttherelativeimportanceofthecriteriainthesettoassessPEpractices.

Keyoutcomes:

TheoutcomesofthethreeDelphigroupsshowedsomesimilarities.Someconvergencecanbeseenatthelevelofthemesasaimsandobjectives,andselectingtherighttargetpopulationtoengagewith scored as top criteria, and sustainabilitywhich scored the lowest in twoDelphigroups (SRP andDCT),while the thirdDelphi group (ED) droppedsustainability in its finalmeeting.

Some criteria were debated more than others due to differences in interpretations of theterminology.Forexample,conflictofinterest(COI)wasunderstoodas‘managing’,‘disclosing’or‘avoiding’bythedifferentstakeholders.Additionally,somebiasininterpretationcouldhavealsocomeduetotheEUandUSperspective.1 PARADIGM is an Innovative Medicines Initiative (IMI) - a public-private partnership co-funded by the European Commission and EFPIA (the European Federation of Pharmaceutical Industries (and Associations). For more information about IMI projects, please visit: https://www.imi.europa.eu/about-imi. For more information about PARADIGM, please visit www.imi-paradigm.eu 2 To learn more about WP1’s online survey and focus group consultations, please see the report: https://imi-paradigm.eu/wp-content/uploads/2019/03/D1.1-Survey-analysis-report.pdf 3 The Delphi methodology originates from the US and is meant to estimate the likelihood of future events. (RAND Corporation, 2019. Internet source, available here: https://www.rand.org/topics/delphi-method.html). In health research, it is widely used as a tool to obtain consensus on group opinion (Trevelyan, 2015. European Journal of Integrative Medicine. Available here: https://www.sciencedirect.com/science/article/abs/pii/S1876382015300160) 4 , “PARADIGM focus[es] on three decision-making points, during which integration of the patient perspective is critical (if not essential) for the medicine lifecycle” as requested by the IMI call (PARADIGM, 2018). More information here: https://www.imi.europa.eu/projects-results/project-factsheets/paradigm.,

9


TheseresultswillbeusedinotherPARADIGMworkpackages(i.e.WP2)inordertoco-createquestions for thegapanalysis tool. Inaddition, theauthorsproposeaway the resultsof theDelphi couldbeoperationalizedasa tool toassessPEpractices ineachof the three specificdecision-making points to determine whether they meet the expectations for a good PEpractice.

10


2. Introduction

PARADIGM, which stands for Patients Active in Research and Dialogues for an ImprovedGenerationofMedicines,isanEUprojectfundedbytheInnovativeMedicinesInitiative(IMI)andtheEuropeanFederationofPharmaceuticalIndustriesandAssociations(EFPIA).Itisa30-monthprojectthatinvolves34partners.ItsmissionistodevelopaPatientEngagement(PE)framework that allows structured, meaningful, sustainable and ethical patient engagementfocused on the three key decision-making points in the lifecycle of amedicine, i.e. researchpriority setting, design of clinical trials and early dialogue with regulators and HealthTechnology Assessment (HTA) bodies. The framework will include recommendations onprocesses,tools(withthecreationoftemplates)andmethodstomeasureanddemonstratetheaddedvalueofinnovativeandeffectiveapproachestopatientengagement.

This work is part of work package 1 (WP1: Defining stakeholders’ preferences, needs andexpectations) deliverables and is a continuation of the online survey and focus groupconsultations created in 20185 that intended tomap out the range and frequency of needs,expectations and desired outcomes for PE. To achieve a consensus for stakeholders’expectationsandneeds,aDelphimethodologywaschosentoaddressthelevelsofspecificityandin-depthinformationobtainedfromthesurveyandfocusgroups.TheDelphiexpertpanelsdetermined which priority and relative value (weights) each need and expectation for PEwouldgetineachofthethreedecision-makingpoints;howtheyrelatetoeachothertoform“goodpractice”;andwhicharemorelikelytopredictdesiredoutcomes.

3. Methodology

The Delphi methodology (Fig. 1) was used to firstly prioritize the stakeholders’ needs andexpectationsandthentoturnthemintoasetofcriteriatoassessPEpracticesandprocesses.

TheDelphiprocess isaconsensustechnique,widelyused inhealthcareresearch6 thatmeetsthe requirements of scientific methods7. This technique helps to identify and prioritise theexpectations and preferences regarding a specific issue. The results obtained from theinteraction of a structured group of experts aremore solid than those from individuals8. Inusingthistechnique,allparticipantsacceptoratleasthavenostrongdisagreementsregardingthe criteria and categories9 developed. Within PARADIGM, the Delphi modified technique

5Formoreinformation,see thereportfromthesurveyanalysishere:https://imi-paradigm.eu/wp-content/uploads/2019/03/D1.1-Survey-analysis-report.pdf6EubankBH,MohtadiNG,LafaveMR,WileyJP,BoisAJ,BoormanRS,etal.UsingthemodifiedDelphimethodtoestablishclinicalconsensusforthe diagnosis and treatment of patientswith rotator cuff pathology. BMCMedResMethodol [Internet]. 2016;16(1):1–15. Available from:http://dx.doi.org/10.1186/s12874-016-0165-87HutchingsA,RaineR,SandersonC,BlackN.Acomparisonofformalconsensusmethodsusedfordevelopingclinicalguidelines.JHealServResPolicy.2006;11(4):218–24.8 BlackN, MurphyM, LampingD,McKeeM, Sanderson C, Askham J, et al. Consensus developmentmethods: A review of best practice increatingclinicalguidelines.JHealServResPolicy.1999;4(4):236–48.9CriteriarefertotheattributesofPEpractices.Categoriesareelementsthatdescribethecharacteristicsandnuanceofeachcriterioninwhich

11


gatherstherelevantissuesinPEwithineachofthethreedecision-makingpointsinmedicinesdevelopment.

Figure1:OverviewofthePARADIGMDelphiprocess

This technique uses qualitative and quantitative data sequentially by combining two onlinesurveying rounds to elicit responses in a systematic manner, and a subsequent meeting inperson that allows expert panels to interact, discuss, debate and justify their viewpoints.Followingthetwoon-linesurvey,afacetofacemeetingwasheldtoachieveconsensusonasetofcriteriatoassessPEpracticesinmedicinesdevelopment.

Usingthis technique,expectationsandpreferences forPEcanbe identifiedandprioritisedinthe three specific points in time. The technique is based on the assumption that theconvergence of “good practice” elements and their relative value, steered by structuredinteractionamongagroupofindividualexpertsonPEprocesses,yieldsmoreaccurateresultsthan those from individuals; thus the consensus produced ismore likely to lend credibilitywithineachofthedecisionmakingpointinmedicinedevelopment.

Procedure

1-DevelopingDelphiQuestionnaires

First,theDelphiquestionnairesweredevelopedforthetwoonlinesurveyingrounds(Fig.1).Aseparatequestionnairewascreatedforeachofthethreedecision-makingpoints.Thesequestionnaireswerebuiltonthebasisoftheinformationobtainedfromvarioussources:

- ThepreviousworkcarriedoutwithinWP1:

• ResultsfromtheSurveyanalysisreportonstakeholder-specificpreferences,needsandexpectationscarriedoutwithinWP1(task1.1)

isincluded

12


• Three focus Groups by Alzheimer Europe, San Juan de Deu (paediatricgroup)andHTAbodies.

• Patient Focus Medicine Development (PFMD) PE Quality Guidance(https://patientfocusedmedicine.org/)

• Literature review fromNational,Europeanand Internationalpublic andprivateinstitutionsandorganisations,academics,regulatorybodies,HTAbodies,patient’sorganisationsetc.ThetypesofdocumentsconsultedandreviewedwerethoserelatedtoPEinthethreedecision-makingpoints10.

Followedthesearch,allgatheredinformationwasanalysed;thoseelementsdeemedrelevantfor the questionnaire were retrieved and organized into criteria with their categories, toanticipate the potential structure of the questionnaires for assessing PE practices11. ThePARADIGMconsortiumwasinvolvedthroughoutthedevelopmentofthequestionnaires.

2-Experts’recruitmentforthethreeDelphipanels

The recruitment process began (see Figure 1) at the same time as the questionnaireswerebeing constructed. The objective was select participants who share and bring a variety ofperspectivesandexpertiseandhavethepotentialtoproviderelevant,keyandrichinformationaboutPEforeachoftheDelphi.Toselecttheexperts,firsttheDelphiteam(WP1leadersandIACS) defined stakeholder and participant profiles for each panel. Themain requirement toparticipate was to have recognised expertise, experience and knowledge within their topicgroup12.Additionallytheselectioncriteriatookintoaccountthetypeofstakeholders,aswellascountryandgenderdiversity,toavoidbiasandsupportglobalapplicability.

For the recruitment process (Figure 2), the PARADIGMconsortiumwas invited to propose candidates fromwithin oroutside the consortium. Members of the PARADIGMInternational Liaison Group (PILG) were also invited toparticipate.Toensureadequaterepresentativenessofviewsaswell as to complement the level of expertise in the expertpanels,theDelphiteamrecruitedotherrelevantexpert(outsidethe Consortium) whose profile was still missing ensurerepresentativenessinthepanelgroups.

Figure2:Expertsrecruitmentandoutreachprocess

10Seeannex1forthelistofsources11Seeannex2forthequestionnaires

12SeeAnnex3forthepanelstakeholderprofile

13


3-Delphifirstandsecondonlineround(R1andR2)

Thefinishedquestionnaireswerecreatedanddistributedonanonlineplatformandaccesstoitwasgiventoexpertpanellistsindividually(seefigure1).Thisplatformprovidesparticipantsthesurveyinformationinreal-time(i.e.howmanyhavealreadyscoredeachcategoryandthecorrespondingmedianscore).Duringthesurveyperiods,thesystemalloweduserstostopandresumethetask(includingmodifyingthepreviousresponses)asmanytimesastheyneeded.Participants could also add comments and suggestions for the questions presented in theplatforminbothrounds.Thesecommentswereusedtopreparethefinalmeetingandsuggestto the experts possible merging and rephrasing of some of the criteria and categories. Aglossarywasincluded,tohelpexpertsunderstandtheconceptsusedinthequestionnaires13.

During the first and second rounds, experts had to rate each question based on theirknowledgeandexpertiseinPE.Concretelyduringthefirstround,expertswereaskedtoassesshowrelevantwaseachquestionwhenassessingPEpracticesusingaLikert-scale14[from1-3(not relevant at all) to 9 (highly relevant)]. The Mean, Median, Coefficient Variation (CV),Interquartile range (IQR), Quartile 1 (Q1) were calculated along the vote distribution.Agreementwasreachedandkeptforprioritysettinginthesecondroundforthosequestionswhen Q1≥7, IQR ≤ 1, CV <20%; andwhen themedian and the votes fittedwithin the samebracket (agreement converged around ‘relevant’) by the experts. Questions were droppedwhenQ1≤6, IQR≥2,CV<20%andtherewasagreementwhenthemedianandvotes fittedwithin the irrelevanceornot clearly relevant bracket.The restof thequestions thatdidnotreachanyagreementwerekeptforreassessmentforthesecondround

Forthesecondround,participantswereaskedtoindividuallyre-scorethecategories.Inthisround,questionsthatexpertshadagreedonasrelevantforassessingPEpracticesduringthefirst round were again presented for prioritisation (from 1=lowest priority to 9= highestpriority). The analysis focused on the Mean, Median, Coefficient deviation (CV), andInterquartilerange(IQR),Quartile1(Q1)aswellasthevotedistributiontodeterminewhetherexperts agreed on the level of priority (1-3 low priority; 4-6 moderate priority; 7-9 highpriority).Questionsthatreachednoagreementinthefirstroundwerepresentedagaininthesecondroundtotheexpertstore-rankusingtheproposedrelevancescalefromthefirstround(from1=’notrelevantatall’to9=‘highlyrelevant’).

4-Meetinginperson

After the twoonline survey rounds finished,only thoseexpertswhocompleted the first andsecondroundwereeligibleandinvitedtoparticipateinthethirdroundmeetinginBrusselstoreachconsensusaboutthefinalcriteriatoassessPEpractices.Therewasonespecificmeetingorganised foreachDelphipanelgroup. The format,procedureandmethodologyofall threemeetingswereidentical.

13SeeAnnex4fortheglossary14SeeAnnex4fortheglossary

14


A facilitator (from the IACS who was not part of the expert panel), helped conduct thediscussionsfollowingastructuredconsensusmethodology.Tworapporteurs(fromIACSwhowerenotpartof thepanelofexpertseither)providedsupport inrecordingvotingprocessesandmodificationsinphrasingandallocationaccordingtoexperts’consensus.Duringthegroupdiscussion,expertswereabletodiscussandelucidatetheirpointofview.

Theobjectivesofthesemeetingswereto

- identifysimilaritieswithinthedifferentcriteriaandcategories,

- resolveeventualredundancies,decidethefinalsetofcriteriaandcategories,

- allocatethecorrespondingweighttoeachcriteriaandcategory.

Inthefirstpartoftheprocess,questionsprioritisedduringthesecondroundwerepresentedto the experts; those questions that failed to reach agreement during the first and secondrounds were also presented to experts who were asked to decide if the questions wererelevanttoassesspracticesofPE,andwhethertokeepordropthem.Merging,rephrasingandreallocatingthecategoriesacrosseachcriterionwereallowedatthisstage.Oncefullconsensuswasreached,thepanelspenttheremainderofthemeetingvoting.

In thesecondpart, theexpertsvotedonthe finalhierarchyof theelements for thedecision-makingpointtoobtainrelativeweightsfortheconsolidatedcriteriaandcategories.

Expertsproceededtoprioritisethecategoriesbyweightingthembysharing100pointsamongthem. The final weight for each category was calculated by averaging total points by thenumberofvoters(dividingtotalpointsbynumberofexpertsandmultiplyingby100,sothesumofcategories’weightwithinacriterionwasalways100).

Theexpertswerethenaskedtoprioritiseandweightthecriteriainthesameway-bysharing100pointsamongallcriteria.Inbothcases,theprocesswasanonymousandprioritisationwasconductedusingadocumentthatincludedallfinalcategoriesandcriteriapreviouslyagreedonbytheexperts.

TheexpertsvotedonthefinalhierarchyofelementsofgoodPEpracticesforthecorrespondingdecision-makingpoint

4. Results4.1. RepresentativenessofthethreeDelphiExpertPanels

Thecompositionof theDelphi expertpanels isdescribed inFigure3.Themainrequirementwasthat theparticipantsholdrelevantknowledgeandexperience in the fieldofPEtheyareinvolvedintohavethepotentialtoproviderelevantinformationaboutpatientengagement.

15


Partic ipants Initially Round 1

Round 2

Round 3 F2F Profiles in the F2F meeting

DE LPHI

S etting research priorities

24 20 16 10

Patient/Patient advocate/ Patient representative, Regulator,Pharma company, professional, Research Funder (public orprivate), Healthcare professionals, Academic, researcher,other

Design of clinical trials 31 29 26 18

Patient/Patient advocate/ Patient representative, Academic,researcher, Healthcare professionals , Pharma companyprofessional, HTA Body, Research Funder (public orprivate), other

Early Dialogues

with HTA and Regulators

26 20 18 12Regulator, Pharma company professional, Academic,researcher, Patient/Patient advocate/ Patient representative,HTA Body, other

TOTAL 81

Figure3:RepresentativenessoftheDelphiexpertsgroups

Nospecific informationwas collected regarding the specific reasons for thedropoutsduringtheprocess. However, somereportedtobeunable toattend themeeting inBrusselsdue toworkcommitments. Someothersinformedthatthereasonfortheirdroppingoutwasduetotheir institution rules and regulations; they were not allowed to attend any meeting withprivatecompaniespresenttoavoidpotentialconflictofinterest.

The reliability of the results could be compromised if many experts drop out of the studybeforeitiscompleted,puttingatrisksomestakeholders'representativeness.Accordingly,theDelphi team analysed the dropouts in each of the Delphi rounds to check for stakeholders’participation tomake sure that representativenesswas not compromised. It was concludedthat,despitethedropouts,responseratehasremainedhighandthedropoutsdidnotaffectanystakeholdergroups’representativeness.ThusthefinalresultsoftheDelphicanbeconsideredvalidwiththeexistingparticipation.

4.2. DelphionPatientEngagementinSettingResearchPriorities

ExpertpanelfortheDelphiinPEinSettingResearchPriorities

Thesocio-demographiccharacteristicsof therespondents in thisDelphi for thethreeroundsaredisplayedinFigure4.

Figure4:DemographicoftheDelphiexpertsinthegroupforsettingresearchpriorities

16


Age (years)

Figure 4: Socio-demographic characteristics of the participants for the Delphi in SRP

Round 1

Gender

Women: 13 (65%) Men: 7 (35%)

Nº participants invited

24

Total responses

20 (84%) < 25: 1 (5%)35-44: 4 (20%)45-54: 10 (50 %)55-64: 4 (20 %) >65 : 1 (1 %)

Round 2 Women: 11 (68,7%)

Men: 5 (31,2%) 20 16 (80%) < 25: 1 (6,25%)35-44: 2 (12,5%)45-54: 9 (56,2 %)55-64: 3 (18,7 %) >65 : 1 (6,2 %)

Round 3F2F meeting

Women: 6 (60%) Men: 4 (40%)16 10 (62,5%) < 25: 1 (10%)

35-44: 2 (60 %)55-64: 1 (10 %)

DELPHISetting Research Priorities

Attendance

Country origen: Denmark, Spain, Germany, Ireland, Italy, Netherlands, USA, France, Czech Republic and United Kingdom

Round1(Online)–RelevanceAssessment

Thefirstroundlastedthreeweeks(November12th2018–December3rd2018).

Fromtheinitial56categories[49categoriesand7PEimpact(PEI)sub-categories]thatmadeupthelisttobeassessed,agreementwasreachedon20categories(17categories+3PEIsub-categories,(table2inAnnex5).Noconsensuswasreachedon32categories(28categoriesand4PEIsub-categories),(table3inAnnex5)and4weredropped,(table4inAnnex5).

Round2(Online)–RelevanceAssessmentandPrioritization

Thesecondroundlastedthreeweeks(January1st2019–January18th2019).

Regardingthe32categories(28categories+4PEIsub-categories)onwhichnoconsensuswasreachedinthefirstround,7(5categories+2PEIsub-categories)weredeemedrelevant(table5inAnnex5).

Asfortherestofthecategories,in14categories(13categories+1PEIsub-categories)stillnoconsensuswasreachedaboutthelevelofrelevance,sotheywereretainedfordiscussionattheface-to-facemeeting(table6inAnnex5)and11(10categories+1PEIsub-categories)weredropped(table7inAnnex5).

17


Figure 5: Socio-demographic characteristics of the participants for the Delphi in DCT

Round 1

Gender

Women: 22 (75,8%) Men: 7 (24,1%)


31

Total responses

29 (93,5%)

< 25: 1 (3,4%)25-34: 2 (6,8%)35-44: 5 (17,2%)45-54: 18 (62,06%)55-64: 2 (6,8%) >65 : 1 (3,4%)


Men: 6 (23,07%) 29 26 (89,6%)

Round 3F2F meeting

Women: 6 (60%) Men: 4 (40%)26 18 (69,2%)

DELPHIDesignig Clinical Trials

Attendance

Country origen: Germany, Belgium, Spain, Austria, Denmark, Canada, Netherlands, USA, France and United Kingdom

< 25: 1 (3,8%)25-34: 2 (7,6%)35-44: 5 (19,2%)45-54: 17 (65,3 %)55-64: 1 (3,8%)

Age (years)

< 25: 1 (5,5%)25-34: 1 (5,5%)35-44: 4 (22,2%)45-54: 11 (61,11%)55-64: 1 (5,5%)

Fromthe20categories(17categories+3PEIsub-categories),agreedasrelevantinRound1,14were considered a high priority and 5were consideredmediumpriority and 1was lowpriority onRound2. Theywere kept for discussion at the face-to-facemeeting. The scoresobtainedforeachcategoryandthecorrespondingcriteriaaresummarizedintable8inAnnex5.

Round3(Meetinginperson)–DiscussionandFinalPrioritization

PEinSettingResearchPrioritiesExpertPanel

Theexpertpanel tookpart in round3meeting in twohalfdays inBrussels.The final setofcriteriawiththeirweightsrecommendedforassessingPEprocessinsettingresearchprioritiesaredetailedinTable9inAnnex5.

4.3. DelphionPatientEngagementinDesigningClinicaltrials

ExpertpanelfortheDelphiinPEinDesigningClinicalTrials

Thesocio-demographiccharacteristicsof therespondents in thisDelphi for thethreeroundsaredisplayedinFigure5

18


Figure5:DemographicoftheDelphiexpertsinthegroupfordesigningclinicaltrials



Fromtheinitial58categories[48categories+10PEimpact(PEI)sub-categories]thatmadeup the list to be assessed, agreement was reached on the relevance of 26 categories (21categories+5PEI subcategories), (table10 inAnnex6).Noconsensuswas reachedon22categories(19categories+3PEIsub-categories),(table11inAnnex6)and10(8categories+2PEIsub-categories)weredropped,(table12inAnnex6).



Forthissecondround,expertssuggestedincorporatingtwonewcategories(1categoryundertheCriterion7:Involvementandparticipationand1asPEIsub-category;bothareindicatedinthe tables) as they considered important and they were not initially included in thequestionnaireofthefirstround.

Regardingthe22categories(19categories+3PEIsub-categories)onwhichnoconsensuswasreached in the firstround,16 (13categories+3PEI subcategories)weredeemedrelevant(table13inAnnex6).

Asfortherestofthecategories,in2categoriesand1ofthenewcategoriesstillnoconsensuswasreachedaboutthelevelofrelevance,sotheywereretainedfordiscussionattheface-to-facemeeting(table14inAnnex6)and4categoriesweredropped(table15inAnnex6).

Fromthe26categories(21categories+5PEIsub-categories),agreedasrelevantinRound1,12wereconsideredashighpriorityand11wereconsideredmediumpriorityand3waslowpriorityonRound2(table16 inAnnex6). Theywerekept fordiscussionat the face-to-facemeeting.


PEinDesigningClinicalTrialsExpertPanel

Theexpertpanel tookpart in round3meeting in twohalfdays inBrussels.The final setofcriteriawiththeirweightrecommendedforassessingPEprocessinindesigningclinicaltrialsaredetailedinTable17inAnnex6.

4.4. DelphionPatientEngagementinEarlyDialogueswithHTA&Regulators

ExpertpanelfortheDelphiinEarlyDialogueswithHTAandRegulators

19


Figure 6: Socio-demographic characteristics of the participants for the Delphi in ED with HTA and Regulators

Round 1

Gender

Women: 14 (70%) Men: 6 (30%)


26

Total responses

20 (76,9%)25-34: 1 (5%)35-44: 4 (20%)45-54: 10 (50%)55-64: 5 (25%)


Men: 5 (27,7%) 20 18 (90%)

Round 3F2F meeting

Women: 9 (75%) Men: 3 (25%)18 12 (66,6%)

DELPHIEarly Dialogues with HTA and R

Attendance

Country origen: United Kingdom, Spain, Italy, Canada, France, Greece, Belgium and Sweden

Age (years)

25-34: 1 (5,5%)35-44: 3 (16,6%)45-54: 9 (50%)55-64: 5 (27,7%)

25-34: 1 (8,3%)35-44: 3 (25%)45-54: 4 (33,3%)55-64: 4 (33,3%)

Thesocio-demographiccharacteristicsoftherespondentsinthisDelphi,forthethreerounds,

aredisplayedinFigure6



From the initial 62 categories [51 categories + 11 patient engagement impact (PEI) sub-categories]thatmadeupthelisttobeassessed,agreementwasreachedontherelevanceof30categories (23 categories + 7 PEI subcategories) (table 18 inAnnex 7).No consensuswasreachedon20categories(18categoriesand2PEIsub-categories)(table19inAnnex7)and12weredropped(table20inAnnex7).

Figure6:DemographicoftheDelphiexpertsinthegroupforearlydialogueswithregulatorsandHTA



20


Regardingthe20categories(18categories+2PEIsub-categories)onwhichnoconsensuswasreachedinthefirstround,7(6categories+1PEIsub-categories)weredeemedrelevant(table21Annex7).

Asfortherestofthecategories,in7categories(6categoriesand1PEIsub-category)stillnoconsensuswasreachedaboutthelevelofrelevance,sotheywereretainedfordiscussionattheface-to-facemeeting(table22inAnnex7)and6categoriesweredropped(table23inAnnex7).

Fromthe30categories(23categories+7PEIsub-categories),agreedasrelevantinround1,9were considered a high priority and 16 were considered medium priority and 5 was lowpriorityinthisroundandwerekeptfordiscussioninroundthree(table24inAnnex7).


PEinEarlyDialogueswithHTA&RegulatorsExpertPanel

Theexpertpaneltookpartintheround3meetingintwohalfdaysinBrussels.ThefinalsetofcriteriawiththeirweightrecommendedforassessingPEprocessinearlydialogueswithHTAandRegulatorsaredetailedinTable25inAnnex7.

5. Discussion It isacknowledgedthat thedifferences in theoutcomes(Table1)of thethreeDelphigroupsresult from differences in interpretation of the questions by the expertswithin each group.Thus,all threeDelphioutputsarenotnecessarilydirectlycomparable(i.e.eachexpertpanelgroup and their questionnaires were different for each decision making point), but keysimilaritiesalsoexist.

Table1:SummaryoffinaloutputfromthethreeDelphipanelgroups

Some of the key findings discussed throughout the face-to-face meetings as follows. Someconvergence can be seen in the outputs of the three Delphi groups. For example, Aims and

21


Objectivesofapracticerankedhigh,whichshowsthegeneralagreementbytheexperts thatthiscriterionisrelevantforagoodPEpractice.TheaimsandobjectivesshouldbeagreedonandunderstandablebyallstakeholdersinvolvedinthePEpractice.Atthisstageofthedesign,thePEpracticeshouldalsocontemplatetodefineandagreeontherolesandresponsibilitiesamong all participants and to include mechanism to ensure that that all participantsunderstandeachother’srolesandresponsibilities.

Foralldecision-makingpoints, therewasamajordiscussionabout thedifficulty inselectingtherightpatients(“…involvepatientsasindividualstoonoonlypatients’organisations”),withspecific characteristics and circumstances to engagewith, which reflect the diversity of thetargetpopulation.Establishingaclearmethodtoselectpatientswasalsodeemedrelevanttoensuretheengagementisnotmeaninglessandincomplete.

Establishingaregularfeedbacksystemtoinformpatientsaboutoutcomesorchangesthatmayhappen ineachphaseof the involvementwasdebatedduring themeetings.However, in thediscussion expertspointedout the difficulty sometimesof giving feedbackwithout breakingconfidentiality requirements, due to the nature of some information. Itwas alsohighlightedthat patients should be able to provide feedback on the information received related to theresults, documents, final output, etc. The experts agreed that feedback is essential after thecompletion of the process and that giving regular feedback closes the loop of patients’involvementintheimplementationofapractice.

Likewise, the criterion related to conflict of interest was extensively discussed in the threeseparate meetings. The communication and management of potential conflicts of interest,especiallythosereferredtopatientsandhealthcareprofessionalswaslargelydiscussed.Itwaspointedoutthat“…conflictsofinterestmustbeavoided,notjustmanagedordisclosed”inaPEpractice.However, as this is not always possible; and if the involvementof somehealthcareprofessionalsisavoidedbecauseoftheirconflictsofinterest,expertswithstrongknowledgeinthefieldwouldbeexcluded;therefore,avaluablecontributioncouldbemissed.Thediscussionalso revolved around payments and funding for participants involved in some PE practices.Therewas a diversity ofopinions and views.On one side, some experts considered, neitherpatients nor professionals should collect payments for their participation because of thepotentialconflictsofinterestthatthesepaymentscouldimply.Otherviewsconsideredthatanyactivityshouldberemuneratedtoanystakeholderinvolved(i.e.healthcareprofessionalsandpatients)becausetheyarecontributingwiththeirknowledge,expertise,timeandeffort.Therewasarangeofopinionsdependingonthetypeofstakeholder(privateorpublicsector)andbytheircountryoforigin.

Finally, another key criterion discussed during the meetings was related to sustainability.Although this criterionwas considered relevant by the experts, during the votingprocess itreached the lowest score in theDelphigroupsof SRP andDCT. In theDelphigroupof earlydialogueswithHTAandRegulatorssustainabilitywasincludedinthefirsttwoonlineroundsbut dropped in the finalmeeting. For someHTAbodies it is already very difficult to start aprocesslikePEso”longtermsustainabilityisnotyetconsideredintheearlydialogues”.Thus

22


sustainabilityshouldnotbeanelementtobeevaluatedinPEpracticesinthisdecision-makingpoint.ItwasalsorecognizedthatsustainabilityofPEforvulnerablepopulationsiskeyandthatthesepopulationsareexperiencingresource,processandculturalbarrierstosustainability.

Anothermajor element debatedwithin sustainabilitywas related to fostering allianceswithprivate and/or public institution to facilitate sustainability in the PE practices. There wereopposing points of view; on one side, some experts from public entities or patientsorganisationsseekpartnershipsinordertoensurelong-termandsustainablepractices,whichmight not be so necessary for the private sector (i.e. one explanation for this was thatsometimesthecontinuityofaprojectwasdecidedbytheinstitution’salreadysetplans).

Biases in the Delphi results may come from two known sources. The assumptions thatdefinitions of PE and patient and public involvement (PPI) relate to only English speakinginterpretationswere acknowledged as a caveat. Additionally,many respondents in the DCTDelphigroupwere fromNorthAmerica,hence somebias in interpretationof responsesandweightingsisinevitablefromanEUvs.USperspective.

6. Proposal for Utilising the Criteria

TranslatingDelphioutputsintoatooltoassessimplementedPEpractices

Inthissection,wesuggesthowtheresultsoftheDelphigroupscouldbetranslatedintoatoolthat can be used to assess PE practices to assess and determine whether they meet theexpectationsforagoodPEpracticeinthesespecificdecision-makingpoints.Withthistool,wepropose a way to operationalize the output of this work, namely the co-prioritised set ofminimumcriteriathatderivedfromstakeholder’sneedsandexpectationsforPE.Theresultsofthe assessment could help users to firstly identify successful practices and secondly toreplicatethemintotheirownsetting.

Itshouldbenoted, that theseproposedtoolsarenotwithinthescopeof thisworkandthushave not been further developed. However, aswe have been part of producing the criteria,thesetoolsareoursuggestionofhowthecriteriacouldbeoperationalized.

Howthecriteriahasbeenoperationalized

This proposed tool provides an approximate score for each category (0 to100)within eachcriteria and a final score (0 to100) for the practice. The scores of each categorywithin thecriteriahavebeenweightedaccordinglytotheresultsoftheprioritisationdonebytheDelphipanels.TheseweightsrepresenttheimportancegivenbytheexpertpanelwithregardstothecompletionofeachcriterioninthePEpracticedesignorimplementation.

Toconverttheweightsintoaglobalscore,theweightofeachcriterioniscalculatedbyaddingtheweightofitscategories.

Howtousethesetools

Thetoolusesaweighted5-itemLikert-scaletoassessPEpractices,however,itshouldbenoted

23


that it cannot fully counter the subjectivity of the userwith regards to the interpretationofhoweachcriteriaaddressedinthePEpractice.

When assessing a PEpractice, the user should choosewithin the likert-scale a score from0(minimum score) to 5 (maximum score) based on the degree of fulfilment of the practiceagainsteachofthecategorywithineachcriterionofthetool.Oncetheassessmentprocessisdone,thescoresareaddeduptogiveafinalscorethatcanbeusedtodemonstratehowwellapracticehasbeendesignedorimplementedaccordingtothecriteria.

To establish a threshold thatwill determine if a practice is good or not (based on the finalscore)couldbedecidedbytheuserpriortotheassessmentprocess.

ToaccesstheproposedtoolforPatientEngagementinSettingResearchPrioritiespleasefollowthislink:https://imi-paradigm.eu/wp-content/uploads/2019/06/Assessment_tool_PE-in-Setting-Research-Priorities.xlsx

ToaccesstheproposedtoolforPatientEngagementinDesigningClinicalTrialspleasefollowthislink:https://imi-paradigm.eu/wp-content/uploads/2019/06/Assessment_tool_PE-in-Design-Clinial-Trials.xlsx

To access the proposed tool forPatient Engagement in EarlyDialogueswithHTA bodies andRegulators Tool please follow this link: https://imi-paradigm.eu/wp-content/uploads/2019/06/Assessment_tool_PE-in-Eary-Dialogues-with-HTAR.xlsx

Annex 1. List of sources

AdelphiValues,Patient-CenteredOutcomes

Patientengagement:Thefutureforoutcomesresearchandclinicaltrials?

http://healthdocbox.com/Psychology_and_Psychiatry/68794253-Patient-engagement-the-future-for-outcomes-research-and-clinical-trials.html

AGREENextStepsConsortium

APPRAISALOFGUIDELINESFORRESEARCH&EVALUATION

https://www.agreetrust.org/wp-content/uploads/2013/10/AGREE-II-Users-Manual-and-23-item-Instrument_2009_UPDATE_2013.pdf

BPSU(BritishPaediatricSurveillanceUnit)

PPIGuidanceforresearcherssupportedbytheBPSU

https://www.rcpch.ac.uk/bpsu-patient-public-involvement

CADTH

PatientInputTemplateandIntegratingCompanionDiagnosticAssessmentIntoCADTH’sDrugReviewPrograms(2017)

https://www.cadth.ca/sites/default/files/cdr/cdr-pdf/CADTH-patient-input-template.docx

Health Technology Assessment of Drugs With Companion Diagnostics at CADTHhttps://cadth.ca/sites/default/files/cdr/cdr-pdf/CDx_Process.pdf

Template for Patient Group Input to CADTH Common Drug Review and pan-CanadianOncologyDrugReview

https://www.cadth.ca/pcodr

DurhamUniversityCentreforSocialJusticeandCommunity

ActionCommunity-basedparticipatoryresearchAguidetoethicalprinciplesandpractice

https://www.dur.ac.uk/resources/beacon/CBPREthicsGuidewebNovember20121.pdf

EMA

Stakeholder Engagement report 2017 Patients, consumers, healthcare professionals,

25


academicsandtheirorganisations.EMA2017

https://www.ema.europa.eu/documents/report/stakeholder-engagement-report-2017_en.pdf

Thepatient'svoiceintheevaluationofmedicinesHowpatientscancontributetoassessmentofbenefitandrisk.EMA.18October2013

https://www.ema.europa.eu/documents/report/report-workshop-patients-voice-evaluation-medicines_en.pdf

EMA-Incorporating patients' views during evaluation of benefit risk by the EMA ScientificCommittees.23October2014 https://www.ema.europa.eu/documents/other/incorporating-patients-views-during-evaluation-benefit-risk-european-medicines-agency-scientific_en.pdf

TheroleofpatientsasmembersoftheEMAHumanScientificCommittees

https://www.ema.europa.eu/documents/other/role-patients-members-european-medicines-agency-human-scientific-committees_en.pdf

EPF

PatientInvolvementinHealthTechnologyAssessmentinEurope(ResultsoftheEPFSurvey)

http://www.eu-patient.eu/globalassets/projects/hta/hta-epf-final-report2013.pdf

Fastercures

PatientPerspectiveValueFramework

http://www.fastercures.org/programs/patients-count/patient-perspective-value-framework/

FDA

Patient-Focused Drug Development Guidance Series for Enhancing the Incorporation of thePatient’sVoiceinMedicalProductDevelopmentandRegulatoryDecisionMaking.Guidance1:collecting comprehensive and representative input.https://www.fda.gov/Drugs/DevelopmentApprovalProcess/ucm610279.htm

PEAC Roundtable Scenarios & Questions. Patient Engagement Advisory CommitteeMeeting.October11and12,2017

https://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/PatientEngagementAdvisoryCommittee/UCM579154.pdf

PATIENT-FOCUSEDDRUGDEVELOPMENT PUBLICWORKSHOP ON GUIDANCE COLLECTING

26


COMPREHENSIVE AND REPRESENTATIVE INPUT DISCUSSION DOCUMENTWorkshop Date:December18,2017(FDA)

https://www.fda.gov/drugs/newsevents/ucm574725.htm

HQIP

Patientandpublicinvolvementstrategy2015-2016

https://www.hqip.org.uk/resource/hqip-patient-and-public-involvement-strategy-201516/#.W9hB7WhKiHs

HTAi

PatientGroupSubmissionTemplateforHTAofmedicines(2014)

https://htai.org/interest-groups/pcig/resources/for-hta-agencies-and-policy-makers/

PatientGroupSubmissionTemplateforHTAofnon-medicines

https://htai.org/interest-groups/pcig/resources/for-hta-agencies-and-policy-makers/

RESOURCES TO INVOLVE PATIENTS AND PATIENT GROUPS IN HTA: Patient GroupSubmissionTemplatesGenericpatientsubmissiontemplatesforanHTA,withusefulpromptsinEnglishandFrench(Medicine’sHTA)

https://htai.org/interest-groups/pcig/resources/for-patients-and-patient-groups/

INVOLVE

Publicinvolvementinclinicaltrials:Supplementtothebriefingnotesforresearchers

http://www.invo.org.uk/wp-content/uploads/2012/03/INVOLVEBriefingnotesforresearchersMarch2012.pdf

NIHR-INVOLVE

Beresford P. Beyond the usual suspects, towards inclusive user involvementhttps://www.invo.org.uk/beyond-the-usual-suspects-towards-inclusive-user-involvement/

IOM(InstituteofMedicine)

ClinicalPracticeGuidelinesWeCanTrust

https://www.ncbi.nlm.nih.gov/books/NBK209539/

27


ISPOR

PatientCenteredInterestGroup

https://www.ispor.org/member-groups/special-interest-groups/patient-centered

KCE

Desomer A et al. Use of patient-reported outcome and experience measures in patient inpatientcareandpolicy.Shortreport.KCEREPORT303Cs.2018

https://kce.fgov.be/sites/default/files/atoms/files/KCE_303C_Patient_reported_outcomes_Short_Report_0.pdf

MarylandCenterofExcellenceinRegulatoryScienceandInnovation

(M-CERSI) Assessing meaningful patient engagement in drug development: a definition,framework,andrubric.UniversityofMaryland

https://www.pharmacy.umaryland.edu/media/SOP/wwwpharmacyumarylandedu/centers/cersievents/pfdd/mcersi-pfdd-framework-rubric.pdf

MEDICALDEVICEINNOVATIONCONSORTIUM(MDIC)

Patientcenteredbenefit-riskprojectreport

http://mdic.org/wp-content/uploads/2015/05/MDIC_PCBR_Framework_Web.pdf

NationalHealthCouncil(NHC)

Dialogue/AdvancingMeaningfulPatientEngagementinResearch,Development,andReviewofDrugsSeptember22,2015

https://www.nationalhealthcouncil.org/sites/default/files/PatientEngagement-WhitePaper.pdf

NationalInstituteforHealthResearch

NationalStandardsforPatientInvolvement

http://www.invo.org.uk/wp-content/uploads/2018/06/Public_Involvement_Standards_v1.pdf

ParentProjectMuscularDystrophy(PPMD)

Key considerations for developing & integrating patient perspectives in drug development:examinationoftheDuchennecasestudy.

https://www.bio.org/sites/default/files/BIO_PPMD_Paper_2016.pdf

28


Patient-CenteredOutcomesResearchInstitute(PCORI)

Engagementinhealthresearch

https://www.pcori.org/engagement

https://www.pcori.org/literature/engagement-literature

PatientFocusedMedicineDevelopment(PFMD)

PatientEngagementQualityGuidanceTool [PlanningaPEproject] [AssessinganongoingorcompletedPEproject]

http://patientfocusedmedicine.org/peqg/patient-engagement-quality-guidance-scenario-2.pdf

SpanishMinistryofHealth,ConsumptionandSocialWelfare

Patient Involvement in the Development of Clinical Guidelines. Methodological handbook.(ImplicacióndePacientesenelDesarrollodeGuíasdePrácticaClínica.ManualMetodológico)GrupodetrabajodeimplicacióndepacienteseneldesarrollodeGPC.ImplicacióndePacientesen el Desarrollo de Guías de Práctica Clínica: Manual Metodológico. Ministerio de Sanidad,ServiciosSocialeseIgualdad.InstitutoAragonésdeCienciasdelaSalud-IACS.GuíasdePrácticaClínicaenelSNS:IACSNº2010/01.

http://www.guiasalud.es/emanuales/pacientes/documentos/manual_pacientes.pdf

TheJamesLindAlliance

TheJamesLindAllianceGuidebook

http://www.jla.nihr.ac.uk/jla-guidebook/downloads/Version-8-JLA-Guidebook-for-download-from-website.pdf

Articles

Boutin et al. Culture and Process Change as a Priority for Patient Engagement inMedicinesDevelopment.TherInnovRegulSci.2017Jan;51(1):29–38

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5302122/

Dubois RW, Lauer M and Perfetto E. When is evidence sufficient for decision-making? Aframework for understanding the pace of evidence adoption. Journal of ComparativeEffectivenessResearch2(4),383–391(2013)

https://www.futuremedicine.com/doi/pdfplus/10.2217/cer.13.39

29


Boudeset al.Whatdo stakeholdersexpect frompatientengagement:Are theseexpectationsbeingmet?HealthExpectations.2018;21:1035–1045

https://onlinelibrary.wiley.com/doi/full/10.1111/hex.12797

Sacristanetal.Patientinvolvementinclinicalresearch:why,when,andhow.Doverpress.27April2016Volume2016:10Pages631—640

https://www.dovepress.com/patient-involvement-in-clinical-research-why-when-and-how-peer-reviewed-fulltext-article-PPA#T1

SupleD.eta.Fromtokenismtomeaningfulengagement:bestpracticesinpatientinvolvementinanEUproject.ResearchInvolvementandEngagement.2015;1:5

https://researchinvolvement.biomedcentral.com/articles/10.1186/s40900-015-0004-9

30


Annex 2. Delphi questionnaires

Questionnaire-DELPHIONPEINSETTINGRESEARCHPRIORITIESCriterion1:Aimandobjectives

R1. The aim and objectives of the practice on PE in research priority are agreed with allstakeholders(i.epatients,academics,researchers,pharmaceuticalcompaniesetc)involved.

R2. Theaimandobjectivesof thepracticeonPE inresearchpriorityarerelated topatients’needsandinterests.

R3.TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants.

Criterion2:ContextualizationofthepracticeonPEinResearchPriority

R4. Thepracticeisbasedonanassessmentofthepatients’needsandpreferenceswhowillbeengagedinsettingresearchpriorityprocesses.

R5. There is a clear understanding of contextual factors that could affect the outcomes ofinvolving patients in setting research priority (e.g funders, policy makers, industry,researchpromoters…etc).

R6.Anyexistingpolicydirectives,legal(governancerequirements)and/orregulatoryframeworkabouthowtoengagepatientsinsettingresearchpriorityhavebeenconsidered

Criterion3:Targetparticipantsinvolvedinpatientengagement

R7.Theprofilesofthepatientsengagedinsettingresearchprioritiesreflecttherelevantdiversityof the population targeted. (e.g., gender, socioeconomic status, ethnicity, culturalbackground,geographicallocationetc)andincludesgendersensitiveinformation.

R8. Theprofilesof thepatientsengaged insettingresearchprioritiesareselected torepresentrelevant patients’ circumstances (e.g., former patients, patients who are at risk for adisease, but do not yet have the disease, severity of the disease, patients’ advocates,patients’ representatives, patient organizations, level of experience in setting researchpriority,patient’sscientificknowledge/background).

R9. The practice includes methods to identify individuals who have the necessary scientificbackgroundorexpertise,“lived”experiencetoparticipateinsettingresearchpriority(e.g.,patient’snetworks,healthcareprofessionals’networks,publicadvertiseetc.).

R10. Thepracticesupportspatients toengageand/orreachout tovulnerablegroupsto involvetheminsettingresearchpriorities,aswellastomentorotherpatients.

R11.Thepracticeincludesrelevantpointsofviewotherrelevantstakeholders(e.g.,carers,tutors,researchers,healthcareproviders,decision-makers,policymakers,funders).

R12. In the PE practice, the roles and responsibilities of all participants involved, includingpatients,areclearlydefinedandagreed.

R13.Thereisclearguidanceabouthowtoassignrolesamongallparticipantsinvolvedinsettingresearchpriority.

R14.Thepracticedisplayflexibilityandopen-mindednessinassigningtherolesforthepatients,according to the profile, interest and background of the patient (e.g. as sources of

31


information,asconsultantsorascollaboratingpartnersoftheprocess).

R15.Thepracticeincludesspecifictools(e.g.,rolesandresponsibilitycharts,proceduralguidancedocuments, protocols) andmechanisms (e.g.,meetings,workshops, training sessions), toensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.

Criterion4:Codeofconduct

R16. The practice includes a code of conduct,which clearly states the rules andmechanism ofparticipationinsettingresearchpriorityforallstakeholdersinvolved.

R17.Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessibleway,andadaptedtothetargetpopulationinvolved.

R18. Thepracticecontainsmechanisms(explainedtoallstakeholdersinaclearandaccessibleway) to manage potential conflicts of interest when patients engage with differentstakeholders(e.g.,policiesthatrequirefulldisclosure,transparency,andaccountability).

R19. The practice contains procedures (e.g., personalized meetings, ways of anonymousuncovering) to identify and address discriminatory, coercive, intimidating, and unethicalbehavioursresultingfromtheirparticipationintheresearchprioritiesprocess.

Criterion5:Resources

R20.Allparticipantsinvolvedinthesettingresearchprioritypracticeareinformed,inclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.

R21. The practice includes an equitable financial compensation framework for all participants(e.g., reimbursement of expenses for travel, subsistence, time missed from work,child/elderlycareetc.).

R22.Fundingisallocatedtocoveralltheelementsofthepractice(e.g.,humanresources,materialresources),andisadequatefortheintendedwork.

R23.Theadequateinfrastructuresneededforpatients’andotherstakeholders’involvementsarein place (e.g., technological support, working space, information and communicationtechnologies) adapted to specific circumstances of participants (e.g possible physical,mental,cognitiveoranyimpairmentsetc.)

R24.Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.

Criterion6:Capacitybuilding

R25. The practice identifies the competencies required by all participants involved in settingresearchpriority(e.gknowledgeandskillsuchashowtoperformpatientengagementinsettingresearchpriority,etc).

R26. Thepracticeincludesspecifictrainingprogramsaboutsettingresearchpriorityprocessesand methods to all participants (e.g., written information, in-house training, online,personalisedassistance,andwebinars).

R27. Thetrainingmaterial isadapted,comprehensibleandaccessibletoallparticipantstakingintoconsiderationimpairments,literacylevels,culturalbackgroundandthecircumstancesofvulnerablepatientsinvolvedinsettingresearchpriority.

R28. The training program specifies time when training needs to be provided, duration and

32


frequencyofthesessions(e.g.,ondemand,once,twiceamonth/year).

R29.Thepracticeincludestrainingforpatientsandotherstakeholdersinvolvedindifferentareas(e.g legal and regulatory concepts and challenges in research or deliberative processes,concepts,terminologyetc).

R30. The practice incorporates training/coaching for stakeholders, other than patients (e.gresearchers,healthcareprofessional,fundersetc)regardingtheparticipationofpatientsinresearchprioritysetting.

Criterion7:involvementandparticipation

R31. The type of interaction among different stakeholders (e.g., cooperation, co-creation,information,advice…)isclearlydefined.

R32. Patients receive timely, clear, accessible and appropriate information about theirinvolvement (i.e what will be addressed, what is expected from them, what theirparticipationwillentail,howthepracticewillbeorganisedetc)

R33. The timing requirement of the engagement in setting research priority is appropriatelyplanned, taking into account patient’s needs, and based upon the requirements of theengagementprocess(e.gtimeoftheday,durationetc).

R34. Mechanisms to ensure the meaningful participation and contribution of all participantsinvolved in the PE activity are in place taking in consideration characteristics andcircumstancesofvulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc).

R35.Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.

R36. The information about the outcomes included for each phase of the involvement iscommunicatedinplainandrespectfullanguage,usingaccessible,clearandunderstandableformats, and taking into consideration potential impairments (e.g., impaired vision,blindness,deafnessandcognitiveproblems),lowliteracylevels,culturalbackgroundsandotherrelevantfactors(e.g.age,disease/condition,etc.).

R37.Thepracticeincludesasinglepointofcontactoranamedpersontowhompatientscanreachout to when needed, for information and/or support, throughout their involvement insettingresearchpriority.

R38. Mechanismsare inplace to guarantee allmembershave the same statuswithin theworkgroup,regardlessoftheirexperienceandknowledge.

Criterion8:EvaluationofthePEPracticeinSettingResearchPriority

R39. An evaluation framework is included (structure, process and results) and it enablesregularityevaluation(frequency,timing…).

R40. The evaluation framework is shared and agreed among the participants to fostercollaboration.

R41.Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.

R42.TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinsettingresearchpriority.

33


R43.Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedinthepracticeusingappropriatechannelsandformatssuitedtotheparticipant’scharacteristicsandneeds.

R44.ThepracticeincludesproceduresbywhichtheconclusionsoftheevaluationofthepracticeonPEareused(e.g.,mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorthecurrentonewithacontinuousimprovementprocess).

Criterion9:PatientEngagementImpact

R45. The practice identifies relevant outcomes and impacts regarding the involvement ofpatientsinsettingresearchpriority.

R46.Assesstherelevanceofthefollowingpossibleoutcomesandimpactsforpatientengagementinsettingresearchpriority:

46.1Researchtopicandprioritiesbecomemoreappropriate,basedonpatients’needs

46.2 The product profile (drug, technology) become more relevant and usable forpatients.

46.3Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients.

46.4 Increased probability of ending the study during the research process (reducedfailureoftheproject).

46.5 Researchproposalssubmittedtopublicfundershavemoreoptionstobefinanced(e.g.enhancedcredibility).

46. 6 Resource allocations when engaging patients within industry become moreappropriateandarebasedonpatients’needs.

46. 7 Funded studies will provide more useful information for patients and decision-makingprocesses

Criterion10:Sustainability

R47. The continuation of thepractice is ensured through ownership and it is embedded in theinstitutionororganisation.

R48.Humanandfinancialresourcesrequiredforthelong-termcontinuityofthepracticeonPEinsettingresearchpriorityareidentifiedandsecured.

R49.Thepracticeidentifiesandestablishesallianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensureitscontinuity.

R50. Thereisadisseminationandcommunicationplan/strategiesoftheresultsinPEinsettingresearchprioritythroughculturallyappropriateandaccessiblemechanisms.

Questionnaire-DELPHIONPEINTHEDESIGNOFCLINICALTRIALSCriterion1:Aimandobjectives

D1.TheaimandobjectivesofthepracticeonPEinthedesigningclinicaltrialsareagreedwithall thestakeholders(i.epatients,academics, researchers,pharmaceuticalcompanies,etc.)involved.

D2.TheaimandobjectivesofthepracticeonPErespondtopatient’sneedsandinterests.

34


D3.TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants.

Criterion2:ContextualizationofthepracticeonPEintheDesignofClinicalTrials

D4.Thepracticeisbasedonapre-assessmentofpatients’wantsandneedswhenparticipatinginthedesignofclinicaltrials.

D5. A clear understanding of contextual factors that would affect the outcomes of involvingpatients in the design of clinical trials is demonstrated (e.g. funding, health system andcoverage,characteristicsofthepopulation,socioeconomicenvironment).

D6.Anyexistingpolicydirectives,legal(governancerequirements)and/orregulatoryframework(e.gEMA)abouthowtoengagepatientsinclinicaltrials’designhavebeenfollowed.


D7.Thepatientprofilesengagedinthedesignofclinicaltrialsreflecttherelevantdiversityofthepopulation targeted (e.g., gender, socioeconomic status, ethnicity, cultural background,geographicallocationetc)andincludesgendersensitiveinformation.

D8.Thetargetparticipant'sprofilewhoparticipateinthedesignoftheclinicaltrialsrepresentthetargetpopulationthatwillberecruitedtotheclinicaltrials(e.g.,formerpatients,peoplewhoareatriskforadisease,butdonotyethavethedisease,patients'advocates,patients'representatives,patientorganisations,levelofexperienceinthedesignofclinicaltrials).

D9.ThepracticeofPEincludesmethodstoidentifyindividualseligibletoengageinthedesignof clinical trials (e.g., patient's networks, health care professionals' networks, publicadvertiseetc).

D10.PracticesonPEinthedesignofclinicaltrialsconsidertheinclusionofotherrelevantpointsof view separated to that of the patients (e.g. carers, tutors, research and academia,healthcare professional, pharmaceutical/biotechnology/medical technology industry,policymaker,regulator,HTAagency,funder,payer…).

D11. The role and responsibilities of all participants involved in the design of clinical trials,includingpatients,areclearlydefinedandagreed.

D12. There is clearguidance abouthow toassign rolesamongallparticipants involved in thedesignofclinicaltrials.

D13.Thepracticedisplaysflexibilityandopen-mindednessinassigningtherolesforthepatients,according to theprofileandbackgroundof thepatient (e.g. assourcesof information, asconsultantsorascollaboratingpartnersoftheprocess).

D14.Thepracticeincludesspecifictools(e.g.,rolesandresponsibilitycharts,proceduralguidancedocuments, protocols) andmechanisms (e.g.,meetings,workshops, training sessions), toensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.

Criterion4:CodeofConduct

D15. The practice includes a code of conduct,which clearly states the rules andmechanism ofparticipationindesignofclinicaltrialsforallstakeholdersinvolved.

D16. Effortsaremade toensure thatall individualsinvolvedknowtherulesofparticipation inclinicaltrialsdesign(e.g.,workshops,meetings).

D17.Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearand

35


accessibleway,andadaptedtothetargetpopulationinvolved.

D18. Potential conflicts of interest are addressed andmanaged (e.g., policies that require fulldisclosure,transparency,andaccountability).

D19. The practices contains mechanisms (e.g., personalized meetings, ways of anonymousuncovering) to identify andaddress discriminatory, coercive, intimidation, and unethicalbehavioursresultingfromtheirparticipationduringthedesignoftheclinicaltrialsprocess.

D20.FundingresourcesforthepracticeonPEinthedesignofclinicaltrialsareclearlyexplainedto all stakeholders involvedprior to involvement,andany changes that occurduring thepracticearecommunicatedup-front.


D21. Allparticipants involved inthedesignofclinicaltrialspracticeareinformed, inclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.

D22. An appropriate and equitable financial compensation framework is in place for allparticipants(e.g.professionals,patientsetc.)invitedtoparticipateinthePEpracticeinthedesignofclinicaltrials(e.g.,reimbursementofexpensesfortravel,subsistence,timemissedfromwork,child/elderlycare).

D23.Fundingisallocatedtocoverallelementsofthepractice(e.g.,humanresources,material),andisadequatefortheintendedwork.

D24.Theadequateinfrastructuresneededforpatients’andotherstakeholders’involvementinthedesign of clinical trials practice are in place (e.g., technological support, working space,information and communication technologies) and adapted to specific circumstances ofparticipants(e.gpossiblephysical,mental,cognitiveoranyimpairmentsetc.)

D25.Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.


D26. The practice includes comprehensive and accessible training or induction material andprogramstorespondtotheneedsofthepeopleinvolvedinthedesignofclinicaltrials.

D27. The training program specifies timewhen training needs to be provided, duration andfrequencyofthesessions(e.g.ondemand,once,twiceamonth/year).

D28.Trainingforpatientsincludesareas,suchaslegalandregulatoryconceptsandchallengesinclinicaltrialsordeliberativeprocess,concepts(e.gbasicstatisticsetc),terminologyetc.

D29. PracticesonPE indesignofclinical trials incorporatetraining/coaching forstakeholders,otherthanpatients,regardinghowtoincorporatepatientsinthedesignofclinicaltrials.

Criterion7:Involvementandparticipation.

D30. The type of interaction among the different stakeholders (e.g., cooperation, co-creation,information,advice…)isclearlydefined.

D31. Consideration has been given to specific participants’ circumstances and characteristicslinkedtobutnotlimitedtopossiblephysicalormentalimpairments,culturalbackground,age and other relevant features (e.g., use of language, format of meetings, the venue,informationprovided).

36


D32. The timing requirementof the engagement in thedesignof clinical trials is appropriatelyplanned,takingintoconsiderationpatient’sneeds,andbasedupontherequirementsoftheengagementprocess(e.gtimeoftheday,durationetc).

D33. Mechanisms to ensure the meaningful participation and contribution of all participantsinvolved in the PE activity are in place taking in consideration characteristics andcircumstancesofvulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc).

D34. Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.

NEWCATEGORY:WhenengagingpatientsintheDesignofclinicaltrials,patient-orientedoutcomesareobtainedthroughvalidinstruments

D35. The information about the outcomes included for each phase of the involvement iscommunicated in plain and respectful language, using accessible, clear andunderstandable formats and taking into account potential impairments (e.g., impairedvision, blindness, and deafness and cognitive problems), low literacy levels, culturalbackgroundsandotherrelevantfactors(e.g.age,disease/condition,etc.).

D36. Asinglepointofcontact fororanamedpersontowhompatientscanreachout towhenneeded for information and/or support, throughout their involvement in the design ofclinicaltrials.

D37. Mechanismsare inplace to guarantee allmembershave the same statuswithin theworkgroup,regardlessoftheirexperienceandknowledge.


D38. An evaluation framework is included (structure, process and results) and it enablesregularityoftheevaluation(frequency,timing…)

D39. The evaluation framework is shared and agreed among the participants to fostercollaboration.

D40.Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.

D41. The evaluation outcomes are linked to the aimand objectives of the PE practice in thedesignofclinicaltrials.

D42.Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedinthepracticeusingappropriatechannelsandformatssuitedtotheparticipant’scharacteristicsandneeds.

D43.ThepracticeincludesproceduresbywhichtheconclusionsoftheevaluationofthepracticeonPEareused(e.g.,mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorthecurrentonewithacontinuousimprovementprocess).


37


D44.Thepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinthedesignofclinicaltrials.

D45.Assesstherelevanceofthefollowingpossibleoutcomesandimpactsforpatientengagementinthedesignofaclinicaltrial:

D45.1Improvedrecruitmentofstudyparticipants

D45.2Increasedretention(i.elowerdropout)ofstudyparticipants

D45.3Increaseddiversityofstudyparticipants

D45.4Betteradherencetotheresearchprotocol,lessamendments

D45.5Improvedstudyparticipants’trialexperience

D45.6Fasterstudycompletionasaresultofimprovedrecruitmentandadherencetotheprotocol

D45.7Decrease costs as a result of improved recruitment, retention and positivetrialexperience

D45.8Moreappropriate,inclusive,sensitiveandethicaltrialdesign

D45.9 More appropriate wording and timing of research instruments (e.g.questionnairesandinterventions)

D45.10Improvedpatientinformationaccessibility(e.g.laysummaries,informationsheets,consentform,recruitmentmaterial)

NEW CATEGORY: Identification clinical trial endpoints that better reflect functionaloutcomes/benefits for patients from using a given therapy compared totraditional/standardregulatoryendpoints


D46. The continuation of the practice is ensured through ownership and it is embedded in theinstitutionororganisation.

D47.Humanandfinancialresourcesrequiredforthelong-termcontinuityofthepracticeonPEinthedesignofclinicaltrialsareidentifiedandsecured.

D48. ThepracticeonPE in thedesignof clinical trials identifies andestablishesallianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensureitscontinuity,aslongasconfidentialityisrespected.

D49. There is a dissemination and communication plan/strategies of the results in PE in thedesignofclinicaltrialsthroughculturallyappropriateandaccessiblemechanisms.

Questionnaire-DELPHIONPEINEARLYDIALOGUES

Criterion1:Aimandobjectives

38


E1. The aim and objectives of PE in Early Dialogues reflects patients’ needs, experiences andexpectationsintheearlydialoguesprocess.

E2. Theaimandobjectivesof thepracticeonPEinearlydialoguesareco-designedandagreedwithallthestakeholdersinvolved.

E3. The aim and objectives of PE in Early Dialogues are clear and understandable for allparticipants.

Criterion2:ContextualizationofthepracticeonPEinEarlyDialogues

E4. In the practice on PE in early dialogues, the process is based on the assessment of thepatient’spreferencesandneeds.

E5.Aclearunderstandingofcontextualfactorsthatinformtheprocessofpatientengagementisdemonstrated so that the quality of dialogue when involving patients in early dialogueswithHTAsandregulatorsisimproved.

E6. Anyexistingpolicy, legal (governancerequirements)andregulatory framework(e.gEMA)about how to engage patients in early dialogues with HTA and regulators have beenfollowed.


E7. The profiles of patients engaged in earlydialogueswithHTA and regulators reflect therelevantdiversityofthepopulationtargeted.(e.g.,gender,socioeconomicstatus,ethnicity,culturalbackground,geographicallocationetc)andincludesgendersensitiveinformation.

E8. Patients engaged in early dialogues with HTA and regulators are selected to representrelevant patients’ circumstances (e.g., former patients, patients who are at risk for adisease, but do not yet have the disease, patients’ advocates, patients’ representatives,patient organizations, level of experience in early dialogues with HTA and regulators,patient’sscientificknowledge/background).

E9. ThepracticeofPEinearlydialoguesincludesmethodstoidentifypatientswhomatchtheclinical criteria needed for a particular dialogue (e.g., patient’s networks, health careprofessionals’networks,publicadvertiseetc.).

E10. Thepracticesupportspatientstoengageand/orreachouttovulnerablegroupstoinvolvetheminearlydialogueswithHTAandregulators,aswellastootherpatients.

E11.Relevantpointsofviewotherthanpatients(e.g.carers,parentsetc.)areconsidered.

E12. Inthepractice,theroleandresponsibilitiesofallparticipantsinvolved,includingpatients,areclearlydefinedandagreed.

E13. Thereisclearguidanceabouthowtoassignrolesamongallparticipantsinvolvedinearlydialogues.

E14.Guidance,trainingadequateisprovidedtoensurethatallparticipantsunderstandtheirownroleandresponsibilities,aswellastheresponsibilitiesandrolesofothers.

Criterion4:CodeofConduct

E15.Thepracticeincludesacodeofconduct,whichclearlystatestherulesofparticipationinearlydialogueswithHTAandregulatorsforallparticipantsinvolved.

39


E16. ThepracticeonPE inearlydialogues includesmechanisms,adequateandadapted to thetargetpopulationtoensurethatallindividualsinvolvedknowtherulesofparticipationinearlydialogueswithHTAandregulators(e.g.,workshops,meetings).

E17.Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessibleway,andadaptedtothetargetpopulation.

E18. The practice on PE in early dialogues incorporates mechanisms (explained to allstakeholdersinaclearandaccessibleway)tomanagepotentialconflictsofinterestwhenpatients engage with different stakeholders (e.g., policies that require full disclosure,transparency,andaccountability).

E19. ThePEpractice containsprocedures (e.g., personalizedmeetings advocacy figure clearlyidentified) to identify and address potential discriminatory, coercive, intimidating, andunethicalbehavioursresultingfromtheparticipationintheearlydialogueprocess.

E20. Funding resources for the practice of PE in early dialogues are clearly explained to allstakeholdersinvolved,priortoinvolvement,andanychangesthatoccurduringthepracticeofPEinearlydialoguesarecommunicatedup-front.


E21. AllparticipantsinvolvedinearlydialogueswithHTAandregulatorsareinformedinclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.

E22.Thepracticeincludesanequitablefinancialcompensationframeworkforallparticipants(e.g.professionals, patients etc.) invited to participate in the early dialogues with HTA andregulators(i.ereimbursementofexpensesfortravel,subsistence,timemissedfromwork,child/elderlycareetc.).

E23.Fundingisallocatedtocoveralltheelementsofthepractice(e.g.,humanresources,materialresources),andisadequatetotheintendedwork.

E24. Adequate infrastructuresneeded forPE inearlydialogues are inplace (e.g., technologicalsupport, working space, information and communication technologies) and adapted tospecific circumstances of participants (e.g. possible physical mental, cognitive or anyimpairmentsetc.).

E25.Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateeffectivepatientengagementimplementationinaclearandaccessibleway.


E26. The practice on PE in early dialogues identifies the competencies required by allstakeholders involved in earlydialogueswithHTAbodies and regulators (e.g knowledgeandskillsuchashowtoperformpatientengagementinearlydialogues,etc.).

E27. The practice on PE in early dialogues includes training programs on HTA/Regulatoryprocesses/clinicaltrialdesignandmethodstopatients(e.g.,writteninformation,in-housetraining,online,personalizedassistance,andwebinars).

E28. The practice includes comprehensive and accessible training or induction material andprogramstorespondtotheneedsoftheparticipantsinvolvedinearlydialogueswithHTAandregulators.

40


E29. The trainingprogramspecifiestimewhentrainingneedstobeprovided,durationandfrequencyofthesessions(e.g.,ondemand,once,twiceamonth/year).

E30. ThepracticeonPE in earlydialogues incorporates training/coaching for all stakeholders,other thanpatients, regarding howto incorporatepatients in earlydialogueswithHTAandregulators.

Criterion7:Involvementandparticipation

E31. The mechanism of interaction among the stakeholder is adapted to their needs. (e.g.meetings,online,skype,etc.).

E32. Patients receive timely, clear, accessible and appropriate information about theirinvolvement (i.e what will be addressed, what is expected from them, what theirparticipationwillentail,howthepracticewillbeorganizedetc)

E33. Thepracticeconsidersspecificpatients’circumstancesandcharacteristicslinkedtobutnotlimited to possible physical or mental impairments, cultural background, age and otherrelevant features (e.g., use of language, format of meetings, the venue, informationprovided).

E34. ThetimingoftheengagementinearlydialogueswithHTAandregulatorsisappropriatelyplanned,basedupontherequirementsofthereviewprocess.

E35. The practice on PE in early dialogues sets mechanisms to ensure a proper deliberativeprocess(e.gmeetings,interviews,focusgroup,workshops…).

E36. A regular feedback system to informpatients about theoutcomes/changes is included ineachphaseoftheinvolvement.

E37. The information about the outcomes included for each phase of the involvement iscommunicatedinplainandrespectfullanguage,usingaccessible,clearandunderstandableformats, and taking into account potential impairments (e.g., impaired vision, blindness,anddeafness),lowliteracylevels,culturalbackgroundsandotherrelevantfactors(e.g.age,disease/condition,etc.).

E38.Thepracticeincludesasinglepointofcontactoranamedpersontowhompatientscanreachouttowhenneeded,forinformationand/orsupport,throughouttheirinvolvementinearlydialogues.

E39. Mechanismsare in place to guarantee allmembers have the same statuswithin theworkgroup,regardlessoftheirexperienceandknowledge.

E40.ThepracticeonPEinearlydialoguesfostersdiscussionandagreementwithpatientsaboutidentifyingandselectingpreferencesabouttheEDprocesswithHTAbodiesandregulators.


E41. An evaluation framework is included (structure, process and results) and it enablesregularityevaluation(frequency,timing…).

E42. The evaluation framework is shared and agreed among the participants to fostercollaboration.

E43.Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.

E44. Theevaluationoutcomesare linked to theaimandobjectivesof thePEpractice inearly

41


dialogues.

E45.Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedinthepracticeusingappropriatechannelsandformatssuitedtoparticipants’characteristicsandneeds.

E46.ThepracticeincludesproceduresbywhichtheconclusionsoftheevaluationofthepracticeonPEareused(e.g.mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess


E47.ThepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinearlydialogueswithHTAandregulators.

E48.AssesstherelevanceofthefollowingpossibleoutcomesandimpactsforpatientengagementinearlydialogueswithHTAandRegulators:

48.1Betterunderstandingoftechnologies’impactinreallifecontextandthequalityoflifeaspects.

48.2Morepracticalandbetterdesigneddevelopmentplanformedicines.

48.3Higheraccuracyinmeasuringneedsandpreferencesofpatients.

48.4BetterqualityoftheevidenceassessedduringthelaterHTAassessment.

48.5Moreusefuladvicegiventoaidcompanydecisions.

48.6Higherrelevanceoftheadvicegiventothelocalcontextofapplication

48.7IncreasedpublicknowledgeandawarenessofHTAandregulatoryprocesses

48.8TransparencyofHTA/regulationprocesses.

48.9Acceptabilityofregulatorydecisions/HTAreportsforstakeholders

48.10 Inclusionofpatients’ relevantoutcomes andend-points in clinical trialsdata collectionandanalysis.

48.11Mutual understanding between all involved stakeholders including the patient communityabouttheregulatoryandHTAprocess.


E49. Thecontinuationofthepracticeisensuredthroughownershipandinstitutionalorpracticeorganizationanchoring.

E50.Humanandfinancialresourcesrequiredforthelong-termcontinuityofthepracticeonPEinearlydialoguesareidentifiedandsecured.

E51.ThepracticeonPEinearlydialoguesidentifiesandestablishesallianceswithotherprivateand/orpublic institutions,citizensandothers, toensure thecontinuityof thePE inearlydialoguesaslongasconfidentialityisrespected.

E52.Thereisadisseminationandcommunicationplan/strategiesoftheresultsoftheprocessofPEinearlydialoguesthroughculturallyappropriateandaccessiblemechanisms.

42


Annex 3. Expert panel stakeholders’ profiles

Stakeholdercategories

Individuals working on orwithexperiencein

Rationaleoftheircontribution

Delphi 1PE inSettingResearchPriorities

ResearchFunders(publicand/orprivate)

*Responsibilityinmanagementand/ordesignofPublicfundingschemes for research(government agencies, publicorganizations, fundingprograms at universities, etc.).*Responsibilityinmanagementand/or design of Privatefunding schemes for research(research and developmentdepartments in companies,scientific societies, charitiesand other privateorganizations,etc.).

Viewsonhowpatientengagementfits into their decisions andprocesses to set priorities for theresearch tobe funded. Suitabilityof engagement practices in place.Howitshapesresults

Regulators Healthauthority atnationalorregional level (end user ofevidence to inform its decisionmaking)

Views on the impact of patientengagement on the researchagenda and how it affects theusability of research findings foradoption and reimbursementdecisions

HTABodies HealthTechnologyAssessment Viewsonhowpatientengagementin research priorities affects thesuitability of available evidencetoassessnewdrugs

Academics,researchers

Conducting and/ or managingresearch

Viewsonhowpatientengagementaffectstheirresearchagenda

HealthcareProfessionals

Healthcare providers(healthcaremanagers (primarycare and hospital),clinicians/health professionalsassociations).

Viewsonhowpatientengagementin research priorities affects theavailability and adoption of newdrugs

Patients/Patientorganizations

Participating (in the role ofpatient) in research prioritiessetting, research projects,researchdesign,etc.

Viewsonhowpatientengagementfits into funders' decisions andprocesses to set researchpriorities. Suitability of

43


engagementpracticesinplace

Delphi 2PE in theDesign ofClinicalTrials

Clinical trialspromoter(pharmacompanies orother)

Designing, preparing andconductingclinicaltrials

Viewsonhowpatientengagementfits into their decisions andprocessestoconductclinicaltrials

Academics,researchers


Viewsonhowpatientengagementaffects their research agenda andthequalityoffindings

Healthcareprofessionals


viewsonhowpatientengagementaffectsnewdrugsavailabilityandadoption

Regulators Clinicaltrialsregulation viewsonhowpatientengagementin design and analysis of clinicaltrialsaffectsthequality(includingethics-wise) and usability offindings


Participating (in the role ofpatient)indesigning,preparingandconductingclinicaltrials.

Viewsonhowpatientengagementfits into researchers and funders'decisions and processes to designclinical trials. Suitability ofengagement practices in place.How it shapes quality andusabilityoffindings

Delphi 3PE inEarlyDialogues

HTAbodies ParticipatinginEarlyDialogues Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagementpractices in place. How it shapesresults

Pharmacompanies,Industry

ParticipatinginEarlyDialogues Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagementpractices in place. How it shapesresults

Regulators Healthauthority atnationalorregional level (reimbursementdecision maker) participating

Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagement

44


inearlydialogues practices in place. How it shapesresults


Participating (in the role ofpatient)inearlydialogues

Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagementpractices in place. How it shapesresults

45


Annex 4. Glossary for the Delphi in PE in Medicine R&D lifecycle

GLOSSARY

Accessible

Adherence

Carer

ClinicalEndpoint[ClinicalTrial]

ClinicalTrialDesign

Culturallyappropriate

CurrentPatient

EarlyDialogue

FormerPatient

HTAAssessment

HTABody

Inclusive

Industry

Likerscale

Outcome

PatientEngagementPractice

PatientOrganisations

PatientRepresentative/PatientAdvocate

Payer

Pointofcontact

PopulationatRisk

Practice

Regulator

RegulatoryFramework

ResearchPrioritiesSetting

Sensitive

Stakeholders

Tutor

46


VulnerablePopulations

Workgroup

AAccessibleEasilyusedoraccessedbypeoplewithspecialneeds.

AdherenceTakingmedications(orothertreatment)exactlyas instructedbyahealthcareprovider(theprescriber).

CCarerAnyone who cares, unpaid, for a friend or family member who, due to illness, disability, amentalhealthproblemoranaddictioncannotcopewithouthisorhersupport.

ClinicalEndpoint[ClinicalTrial]A characteristic or variable that reflects how a patient feels, functions, or survives. Clinicalendpointsaredistinctmeasurementsoranalysisofdiseasecharacteristicsreflectingtheeffectofatherapeuticinterventioninaclinicaltrialorstudy.

ClinicalTrialDesignProcedurebywhichthedetailsofaclinicaltrialaredefined.Theseaspectsincludethedesignoftheprotocol,thediscussionthepotentialburdenthatthepeoplewhowillparticipateinthetrial may experience or the discussion the outcomes of the trial, which could be moreimportanttopatients.

CulturallyappropriateConformingtoaculture'sacceptableexpressionsandstandardsofbehaviourandthoughts.

CurrentPatientA person that has been diagnosed of the condition studied in the practice at stake (SettingResearch Priorities, Designing a Clinical Trial, Early Dialogue) and is currently following atreatment.

E

47


EarlyDialogueEarly scientificdiscussions that can takeplacebetween industry,HTAbodiesandregulators(andinsomecaseswithpayers)todiscussdevelopmentalplansforamedicinalproductandtoensuretheseplansmeettherequirements.

FFormerPatientA person that has received the treatment of the condition studied in the practice at stake(SettingResearchPriorities,DesigningaClinicalTrial,EarlyDialogue),andisnotfollowinganytreatmentcurrently,exceptforfollow-upperiodicreviews.

HHTAAssessmentMultidisciplinary process that summarises information about the medical, social, economic,and ethical issues related to the use of a health technology in a systematic, transparent,unbiased,androbustmanner.Itsaimistoinformtheformulationofsafeandeffectivehealthpoliciesthatarepatientfocusedandseektoachievebestvalue.

HTABodyRegional and national organizations formed by multidisciplinary teams that providerecommendationsonmedicinesandotherhealthtechnologies(HTAassessment),thatcanbefinancedorreimbursedbythehealthcaresysteminaparticularMemberStateorregion.

IInclusiveNotexcludinganysectionofsocietyoranyparty involved.Deliberatelyavoidingusagesthatcouldbeseenasexcludingaparticularsocialgroup.

IndustryPrivate companiesworking in thedevelopmentand/or commercializationofmedicinesandnewhealthtechnologies.

LLikerscale

A Likert scale is a psychometric scale commonly involved in research that employs

48


questionnaires. It is a type of rating scale used to measure attitudes or opinions where,respondentsareaskedtorateitemsonalevelofagreement.

OOutcome

Outcomes refer to decisions made and things produced as a result of patient engagementpractices(includingchangedresearchprioritiesandtrialdesigns).Thisincludestheeffectsonthe development ofmedical products. Outcomes are influenced by context andmechanismsandhaveaninfluenceonimpact.

PPatientEngagementPracticeThe effective and active collaboration of patients, patient advocates, patient representativesand/or carers in a procedure of setting research priorities, designing clinical trials orparticipatinginearlydialogues.Byeffectiveandactivecollaborationsofpatients,wemeanthattheir views, preferences and values are gathered during the practice and are treated in thesamewayasthecontributionsmadebytheotherstakeholdersinvolved.

PatientOrganisationsNot-for profit organisationswhich are patient focused, andwhereby patients and/or carers(thelatterwhenpatientsareunabletorepresentthemselves)representamajorityofmembersingoverningbodies.

PatientRepresentative/PatientAdvocatePatients,carers,parentsortutorsand/orstafffrompatient’sorganizationthatactasaliaisonbetweenthepatientsandotherorganizations,representtheirinterestsandadvocatesfortheirrights.

PayerInsurerorotherorganizationthatpaysforhealthcareservices.

PointofcontactPersonorpersonsthatcanbeapproachedforinformationorassistance.

PopulationatRiskPopulationthatisexposedtodevelopingtheconditionstudiedinthepracticeatstake(SettingResearchPriorities,DesigningaClinicalTrial,EarlyDialogue).

PracticeThe customary, habitual or expected way, method or modality of performing an action. A

49


practice can be the designof the recruitmentprotocol of a clinical trialor the prioritizationexerciseofagivensetofresearchlines,amongothers.

RRegulatorHealthauthorityatnationalorregionallevel.

RegulatoryFrameworkThesetofrulesorordersissuedbyanexecutiveauthorityorregulatoryagencyofagovernmentandhavingtheforceoflaw.

ResearchPrioritiesSettingProcedurebywhichlinesorareasofresearchareidentifiedasapriorityinaresearch-fundingprogram.

SSensitiveCapableofmeasuringorrecordingverysmallchanges

StakeholdersAgencies,organisations,groupsorindividualsthathaverelevantknowledgeand/orexperienceinthepracticeatstake(SettingResearchPriorities,DesigningaClinicalTrial,EarlyDialogue).Theymakeuptheworkgroup.

TTutorThelegalguardianofapatient.

VVulnerablePopulationsThese terms are applied togroups of peoplewho, due to factors usually consideredoutsidetheircontrol;donothavethesameopportunitiesasother,more fortunategroups insociety.Examplesmightincludeunemployedpeople,refugeesandotherswhoaresociallyexcluded.

50


WWorkgroupGroupofstakeholderparticipatingintheclinicaltrialdesignpractice,researchprioritysettingpracticeorintheearlydialogue.

SourcesofInformation

1. MerriamWebsterOnlineDictionary2. Carerstrust3. EURORDISClinicalTrialGlossary4. GlossaryforHIVPrevention,StateofColorado20165. IMIPARADIGM6. EUPATIHTAGlossary7. EMA,StakeholdersandCommunicationsDivision8. OxfordOnlineDictionary9. OECDGlossaryofStatisticalTerms10. CollinsOnlineDictionary11. WHOGlossary12. https://www.statisticshowto.datasciencecentral.com/likert-scale-definition-and-

examples/13. https://en.wikipedia.org/wiki/Likert_scale

51


Annex 5. Results of the Delphi on PE in Setting Research Priorities

ResultsofRound1(Online)–RelevanceassessmentTable2:RelevantcategoriesonPEinSettingResearchPriorities(Round1)

CriteriaID

CriteriaCategory

IDCategory

1 Aimandobjectives 1 TheaimandobjectivesofthepracticeonPEinresearchpriorityareagreed with all stakeholders (i.e. patients, academics, researchers,pharmaceuticalcompaniesetc.)involved.

2 TheaimandobjectivesofthepracticeonPEinresearchpriorityarerelatedtopatients’needsandinterests.

2 Contextualization ofthepracticeonPEinResearchPriority

4 The practice is based on anassessment of the patients’ needs andpreferences who will be engaged in research priority settingprocesses.

3 Target participantsinvolved in patientengagement

7 Theprofilesofthepatientsengagedinsettingtheresearchprioritiesreflecttherelevantdiversityofthepopulationtargeted.(e.g.,gender,socioeconomic status, ethnicity, cultural background, geographicallocationetc.)andincludesgendersensitiveinformation.

8 Patientsengagedprofileinsettingresearchprioritiesareselectedtorepresent relevant patients’ circumstances (e.g., former patients,patientswhoareatriskforadisease,butdonotyethavethedisease,patients’ advocates,patients’ representatives,patientorganizations,level of experience in research priority setting, patient’s scientificknowledge/background).

12 In thePE practice, the roles and responsibilities of all participantsinvolved,includingpatients,areclearlydefinedandagreed.

4 Codeofconduct 16 Thepracticeincludesacodeofconduct,whichclearlystatestherulesandmechanism of participation in setting research priority for allstakeholdersinvolved.

17 The terms and conditions of confidentiality agreements areadequate,written inaclearandaccessiblewayandadaptedto thetargetpopulationinvolved.

18 Thepracticecontainsmechanisms(explainedtoallstakeholdersinaclear and accessible way) to manage potential conflicts of interestwhenpatientsengagewithdifferentstakeholders(e.g.,policiesthatrequirefulldisclosure,transparency,andaccountability).

5 Resources 23 The adequate infrastructures needed for patients’ and otherstakeholders’ involvement is in place (e.g., technological support,working space, information and communication technologies)adapted to specific circumstances of participants (e.g. possiblephysical,mental,cognitiveoranyimpairmentsetc.)

24 Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.

52


6 Capacitybuilding 25 Thepracticeidentifiesthecompetenciesrequiredbyallparticipantsinvolvedinsettingresearchpriority(e.g.knowledgeandskillsuchashowtoperformpatientengagementinsettingresearchpriority,etc.).

7 Involvement andparticipation

32 Patientsreceivetimely,clear,accessibleandappropriateinformationabout their involvement (i.e. what will be addressed, what isexpected from them, what their participation will entail, how thepracticewillbeorganisedetc.)

36 Theinformationabouttheoutcomes includedforeachphaseof theinvolvement is communicated in plain and respectful language,usingaccessible,clearandunderstandableformats,andtakingintoconsideration potential impairments (e.g., impaired vision,blindness, deafness and cognitive problems), low literacy levels ,culturalbackgroundsandother relevant factors (e.g. age,disease/condition,etc.).

37 Thepracticeincludesasinglepointofcontactoranamedpersontowhom patients can reach out to when needed, for informationand/or support, throughout their involvement in setting researchpriority.

8 Evaluation 42 TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinsettingresearchpriority.

9 Patient EngagementImpact

45 Thepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinsettingresearchpriority.

46.1 Research topic and priorities becomemore appropriate, based onpatients’needs

46.2 The product profile (drug, technology) becomemore relevant andusableforpatients.

46.3 Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients.

Table3:NoconsensuscategoriesonPEinSettingResearchPriorities(Round1)

CriteriaID

Criteria CategoryID

Category

1 Aimandobjectives 3 The aim and objectives of the practice on PE are clear andunderstandableforallparticipants.

2 Contextualizationofthepractice on PE inResearchPriority

5 Thereisaclearunderstandingofcontextualfactorsthatcouldaffecttheoutcomesof involvingpatients in research settingpriority (e.g.funders,policymakers,industry,researchpromoters…etc.).

6 Any existing policy directives, legal (governance requirements)and/or regulatory framework about how to engage patients insettingresearchpriorityhavebeenfollowed.


9 Thepractice includesmethodsto identify individualswhohavethenecessaryscientificbackgroundorexpertisetoparticipateinsettingresearchpriority(e.g.,patient’snetworks,healthcareprofessionals’networks,publicadvertiseetc.).

10 The practice supports patients by sharing responsibility ofengaging/reaching out to vulnerable groups to involve them in

53


settingresearchpriorities,aswellastomentorotherpatients.

13 There is clear guidance about how to assign roles among allparticipantsinvolvedinsettingresearchpriority.

14 Thepracticedisplayflexibilityandopen-mindednessinassigningtheroles for the patients, according to the profile, interest andbackground of the patient (e.g. as sources of information, asconsultantsorascollaboratingpartnersoftheprocess).

15 The practice includes specific tools (e.g., roles and responsibilitycharts,proceduralguidancedocuments,protocols)andmechanisms(e.g., meetings, workshops, training sessions), to ensure that allparticipants understand their own and other’s roles andresponsibilities.

4 Codeofconduct 19 Thepracticecontainsmechanisms(e.g.,personalizedmeetings,waysof anonymous uncovering) to identify and address discriminatory,coercive,intimidating,andunethicalbehavioursresultingfromtheirparticipationintheresearchprioritiesprocess.

5 Resources 20 Allparticipantsinvolvedinthesettingresearchprioritypracticeareinformed, in clearandaccessibleway,of the resourcesavailable tosupporttheirtasksduringtheprocess.

21 The practice includes an equitable financial compensationframeworkfor allparticipants(e.g.,reimbursementofexpenses fortravel,subsistence,timemissedfromwork,child/elderlycareetc.).

6 Capacitybuilding 26 The practice includes specific training programs about settingresearch priority process and methods to all participants (e.g.,written information, in-house training, online, personalisedassistance,andwebinars).

27 The trainingmaterial is adapted, comprehensible andaccessible toallparticipantstakingintoconsiderationimpairments,literacylevels,cultural background and the circumstances of vulnerable patientsinvolvedinsettingresearchpriority.

29 The practice includes training for patients and other stakeholdersinvolved in different areas (e.g. legal and regulatory concepts andchallenges in research or deliberative processes, concepts,terminologyetc.).

30 Thepractice incorporates training/coachingfor stakeholders,otherthanpatients(e.g.researchers,healthcareprofessional,fundersetc)regardingtheparticipationofpatientsinresearchprioritysetting.


31 The type of interaction among different stakeholders (e.g.,cooperation,co-creation,information,advice…)isclearlydefined.

33 The timing requirement of the engagement in setting researchpriorityisappropriatelyplanned,takingintoaccountpatient’sneeds,and based upon the requirements of the engagement process (e.g.timeoftheday,durationetc.).

34 Mechanismstoensurethemeaningfulparticipationandcontributionof allparticipants involved in thePEactivity are inplace taking inconsideration characteristics and circumstances of vulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc.).

35 Aregularfeedbacksystemto informpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.

54


38 Mechanisms are in place to guarantee allmembers have the samestatus within the work group, regardless of their experience andknowledge.

8 Evaluation 40 The evaluation framework is shared and agreed among theparticipantstofostercollaboration.

41 Expertise in evaluation process, methods, tools and monitoringsystemareinplacetocreateandpopulatetheevaluationframework.

43 The evaluation outcomes are shared with all the participantsinvolved in the practice using appropriate channels and formatssuitedtotheparticipant’scharacteristicsandneeds.

44 The practice includes procedures by which the conclusions of theevaluationareused(e.g.,mechanismstointegratetheresultsoftheevaluation in the design of the next practice or in the current onewithacontinuousimprovementprocess).

9 Patient EngagementImpact

46.4 Increased probability of success of the study during the researchprocess(reducedfailureoftheproject).

46.5 Researchproposals submitted topublic fundersaremore fundable(e.g.enhancedcredibility).

46.6 Resourceallocationwithinindustrybecomemoreappropriatebasedonpatients’needs.

46.7 Funded studies will provide more useful information for patientsanddecision-makingprocesses

10 Sustainability 47 The continuationof thepractice isensured throughownershipandinstitutionalororganisationalanchoring.

48 Humanandfinancialresourcesrequiredforthelong-termcontinuityof thepracticeonPEin setting researchpriorityare identifiedandsecured.

49 The practice identifies and establishes alliances with other privateand/or public institutions, citizens and others, to ensure itscontinuity.

50 There isadisseminationand communicationplan/strategiesof theresults in PE in setting research priority through culturallyappropriateandaccessiblemechanisms.

Table4:DroppedcategoriesonPEinSettingResearchPriorities(Round1)

CriteriaID

CriteriaCategory

IDCategory


11 The practice includes relevant points of view other relevantstakeholders(e.g.,carers,tutors,researchers,healthcareproviders,decision-makers,policymakers,funders).

5 Resources 22 Funding is allocated to coverall theelementsof thepractice (e.g.,human resources, material resources), and is adequate for theintendedwork.

6 Capacitybuilding 28 The training program specifies time when training needs to beprovided,durationandfrequencyof thesessions(e.g.,ondemand,

55


once,twiceamonth/year).

8 Evaluation 39 An evaluation framework is included (structure, process andresults)anditenablesregularityevaluation(frequency,timing…).

ResultsofRound2(Online)–PrioritizationTable5:RelevantcategoriesonPEinSettingResearchPriorities(Round2)

CriteriaID

Criteria CategoryID

Category

1 Aim andobjectives

3 TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants.

3 Targetparticipantsinvolved inpatientengagement

15 The practice includes specific tools (e.g., roles and responsibility charts,proceduralguidancedocuments,protocols)andmechanisms(e.g.,meetings,workshops, training sessions), to ensure that all participants understandtheirownandother’srolesandresponsibilities.

7 Involvementandparticipation

33 The timing requirement of the engagement in setting research priority isappropriatelyplanned, taking intoaccountpatient’sneeds,andbaseduponthe requirementsof theengagementprocess(e.g. timeof theday,durationetc.).

35 Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.

8 Evaluation 41 Expertiseinevaluationprocess,methods,toolsandmonitoringsystemareinplacetocreateandpopulatetheevaluationframework.

9 PEImpact 46.6 Resource allocation within industry become more appropriate based onpatients’needs.

46.7 Funded studies will provide more useful information for patients anddecision-makingprocesses

Table6:NoconsensuscategoriesonPEinSettingResearchPriorities(Round2)

CriteriaID

Criteria CategoryID

Category

2 Contextualizationof the practice onPE in ResearchPriority

6 Any existing policy directives, legal (governance requirements) and/orregulatory frameworkabout how to engage patients in setting researchpriorityhavebeenfollowed.


13 There isclearguidanceabouthowtoassignrolesamongallparticipantsinvolvedinsettingresearchpriority.

56


5 Resources 20 All participants involved in the setting research priority practice areinformed, in clear and accessible way, of the resources available tosupporttheirtasksduringtheprocess.

21 Thepracticeincludesanequitablefinancialcompensationframeworkforall participants (e.g., reimbursement of expenses for travel, subsistence,timemissedfromwork,child/elderlycareetc.).

6 Capacitybuilding 27 The training material is adapted, comprehensible and accessible to allparticipants taking into consideration impairments, literacy levels,cultural background and the circumstances of vulnerable patientsinvolvedinsettingresearchpriority.

30 Thepractice incorporates training/coachingfor stakeholders,other thanpatients(e.g.researchers,healthcareprofessional,fundersetc.)regardingtheparticipationofpatientsinresearchprioritysetting.


34 MechanismstoensurethemeaningfulparticipationandcontributionofallparticipantsinvolvedinthePEactivityareinplacetakinginconsiderationcharacteristicsandcircumstancesofvulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc.).

8 Evaluation 40 Theevaluationframeworkissharedandagreedamongtheparticipantstofostercollaboration.

43 Theevaluationoutcomesaresharedwithall theparticipantsinvolvedinthe practice using appropriate channels and formats suited to theparticipant’scharacteristicsandneeds.

9 PEImpact 46.5 Research proposals submitted to public fundersaremore fundable (e.g.enhancedcredibility).

10 Sustainability 47 The continuation of the practice is ensured through ownership andinstitutionalororganisationalanchoring.

48 Human and financial resources required for the long-term continuity ofthepracticeonPEinsettingresearchpriorityareidentifiedandsecured.

49 Thepracticeidentifiesandestablishesallianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensureitscontinuity.

50 Thereisadisseminationandcommunicationplan/strategiesoftheresultsin PE in setting research priority through culturally appropriate andaccessiblemechanisms.

Table7:DroppedcategoriesonPEinSettingResearchPriorities(Round2)

CriteriaID

Criteria CategoryID

Category

2 Contextualization ofthepracticeonPEinResearchPriority

5 There isaclearunderstandingofcontextual factorsthatcouldaffectthe outcomes of involving patients in research setting priority (e.g.funders,policymakers,industry,researchpromoters…etc.).

3 Target participantsinvolved in patient

9 The practice includesmethods to identify individualswho have thenecessaryscientificbackgroundorexpertisetoparticipate insetting

57


engagement research priority (e.g., patient’s networks, health care professionals’networks,publicadvertiseetc.).

10 The practice supports patients by sharing responsibility ofengaging/reachingouttovulnerablegroupstoinvolvetheminsettingresearchpriorities,aswellastomentorotherpatients.

14 Thepracticedisplayflexibilityandopen-mindednessinassigningtheroles for the patients, according to the profile, interest andbackground of the patient (e.g. as sources of information, asconsultantsorascollaboratingpartnersoftheprocess).

4 CodeofConduct 19 Thepracticecontainsmechanisms(e.g.,personalizedmeetings,waysof anonymous uncovering) to identify and address discriminatory,coercive, intimidating,andunethicalbehaviours resulting from theirparticipationintheresearchprioritiesprocess.

6 CapacityBuilding 26 The practice includes specific training programs about settingresearchpriorityprocessandmethodstoallparticipants(e.g.,writteninformation, in-house training, online, personalised assistance, andwebinars).

29 The practice includes training for patients and other stakeholdersinvolved in different areas (e.g. legal and regulatory concepts andchallenges in research or deliberative processes, concepts,terminologyetc.).


31 The type of interaction among different stakeholders (e.g.,cooperation,co-creation,information,advice…)isclearlydefined.

38 Mechanisms are in place to guarantee all members have the samestatus within the work group, regardless of their experience andknowledge.

8 Evaluation 44 The practice includes procedures by which the conclusions of theevaluationareused (e.g.,mechanisms to integrate the resultsof theevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess).

9 PEImpact 46.4 Increased probability of success of the study during the researchprocess(reducedfailureoftheproject).

Table8:RelevantcategoriesonPEinSettingResearchPrioritiesandtheirPriority(Round2)

CriteriaID

Criteria CategoryID

Category Priority

1 Aim andobjectives

1 The aim and objectives of the practice on PE in researchpriority are agreed with all stakeholders (i.e. patients,academics, researchers, pharmaceutical companies etc.)involved.

HP

2 The aim and objectives of the practice on PE in researchpriorityarerelatedtopatients’needsandinterests.

HP

58


3 Targetparticipantsinvolvedinpatientengagement

12 In the PE practice, the roles and responsibilities of allparticipantsinvolved,includingpatients,areclearlydefinedandagreed.

HP

4 CodeofConduct 16 Thepracticeincludesacodeofconduct,whichclearlystatestherulesandmechanismofparticipationinsettingresearchpriorityforallstakeholdersinvolved.

HP

17 Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulationinvolved.

HP

18 The practice contains mechanisms (explained to allstakeholders in a clear and accessible way) to managepotential conflicts of interest when patients engage withdifferent stakeholders (e.g., policies that require fulldisclosure,transparency,andaccountability).

HP

5 Resources 23 Theadequateinfrastructuresneededforpatients’andotherstakeholders’ involvement is in place (e.g., technologicalsupport, working space, information and communicationtechnologies) adapted to specific circumstances ofparticipants(e.gpossiblephysical,mental,cognitiveoranyimpairmentsetc.)

HP


32 Patients receive timely, clear, accessible and appropriateinformation about their involvement (i.e. what will beaddressed, what is expected from them, what theirparticipationwillentail,howthepracticewillbeorganisedetc.)

HP

36 The information about the outcomes included for eachphase of the involvement is communicated in plain andrespectful language, using accessible, clear andunderstandable formats, and taking into considerationpotential impairments (e.g., impaired vision, blindness,deafness and cognitive problems), low literacy levels ,cultural backgrounds and other relevant factors (e.g. age,disease/condition,etc.).

HP

8 Evaluation 42 The evaluation outcomes are linked to the aim andobjectivesofthePEpracticeinsettingresearchpriority.

HP

9 PEImpact 45 The practice identifies relevant outcomes and impactsregarding the involvement of patients in setting researchpriority.

HP

46.1 Research topic and priorities become more appropriate,basedonpatients’needs

HP

46.2 The product profile (drug, technology) become morerelevantandusableforpatients.

HP

46.3 Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients.

HP

59


2 Contextualizationof the practice onPE in ResearchPriority

4 The practice is based on an assessment of the patients’needs and preferences who will be engaged in researchprioritysettingprocesses.

MP


7 Theprofilesofthepatientsengagedinsettingtheresearchpriorities reflect the relevant diversity of the populationtargeted. (e.g., gender, socioeconomic status, ethnicity,cultural background, geographical location etc.) andincludesgendersensitiveinformation.

MP

8 Patients engaged profile in setting research priorities areselectedtorepresentrelevantpatients’circumstances(e.g.,formerpatients,patientswhoareatrisk foradisease,butdo not yet have the disease, patients’ advocates, patients’representatives,patientorganizations,levelofexperienceinresearch priority setting, patient’s scientific knowledge/background).

MP

5 Resources 24 Thepracticeincludesguidance,andready-to-usetoolsandtemplates material that facilitates effective patientengagementimplementationinaclearandaccessibleway.

MP


37 Thepractice includesa singlepointof contactor anamedpersontowhompatientscanreachouttowhenneeded,forinformationand/or support, throughout their involvementinsettingresearchpriority.

MP

6 CapacityBuilding 25 The practice identifies the competencies required by allparticipants involved in setting research priority (e.g.knowledge and skill such as how to perform patientengagementinsettingresearchpriority,etc.).

LP

Round3(Meetinginperson)–DiscussionandFinalPrioritizationDiscussionandFinalPrioritizationprocess

Toeasediscussionsatthemeeting,thehighlyprioritiseditemsandnon-consensuscategoriesremainingfromthesecondroundwerefurtherelaboratedbythePARADIGMDelphiteam. Inordertoavoidredundancy,aproposalwasmadetomerge,rephraseorreallocatecategoriesandcriteria.Theywerepresentedtotheexpertpanelandthoroughlydiscussedatthemeeting.Comments provided by the experts in the first and second round were taken intoconsideration.

From the initial 10 thematic criteria, the experts agreed on 8 criteria that consisted of 29categories, and 6 subcategories. The experts rephrased categories in order to avoidredundancyandoverlapping.

Some parts of the discussions are captured below to provide more clarity on the keydiscussionsduringthemeeting.

Overall, experts hold that the disclaimers agreed at the meeting should be captured in thedocumentinaclearandaccessibleway.Besides,itwashighlightedthatitisnecessarytotake

60


into consideration any stakeholder and participant involved along the practice (e.g. patientrepresentatives, researchers, industrypartners, etc.). Furthermore, theyagreedon removingbracketsalongthedocumenttobereadwithease,andbecausetheywerealreadyincludedintheglossaryand/orfootnotes(Annex4).

Experts agreed on merging criterion 1, “Aim and objectives”, criterion 2,“Contextualization of the practice on patient engagement in research priority” andcriterion3,“Targetparticipantsinvolvedinpatientengagement”.Thenewcriterionwasnamed“Keyelementsofthepracticedesign”.Allthecategoriesincludedinthiscriterionwereconsideredbasicandrelevantforthepractice.

Expertspointedoutthatfeaturesapproachedincategories1,2and3overlapped,sotheywerecombinedinanewone.

Category 4was considered relevant but itwas rephrasedwith regard to every stakeholder:researchers,industryandpatients.

Categories7and8weremergedandexpertsconsidered that twonewdefinitionsshouldbeincluded in the glossary: “target population” and “patient’s circumstances”. Additionally,category15wasrephrased,makingiteasytoread,andwithclarificationsaboutsomewords:“specifictools”and“mechanisms”.

Criterion 4, “Code of conduct”, did not experience many changes. The three categoriesincludedinthiscriterionwererephrased.

The expert panel refused the idea of merged criteria 5, 6, and 7. All of them were barelychanged.

Regardingthecategorieswithinthecriterion5,“Resources”, somecategoriesweremergedand rephrased for the criterion to be clearer and more concise. The panel identifiedoverlappingofcategories20and21,whichweremergedtoavoidredundancy.Expertsmadeclearthat“itisnecessaryanequalframeworkandcompensationtosupportalltheparticipantsduringtheprocess”.Concerningcategories23and24,theywererephrased.

Criterion6,“Capacitybuilding”,wasbarelychanged.Expertsdiscussedcategory30andtheyconsideredthatthetrainingisreallyrelevant.Theyputintocontextthatwell-selectedpatientsmustbetrainedbeforetheirparticipation.

Again, categories within criterion 7, “Involvement and participation”, experiencedrephrasing andmerging to avoid repetition. Experts highlighted the need to clarify that thiscriterionreviewstheexpectationsoftheinvolvementinthepractice,notpatientexpectations.

Regarding formercriterion8, “Evaluationof thepractice insettingresearchpriorities”,categories 41 and 42 did not suffer any change. Categories 40 and 43 were rephrased. Inregard to category 43, the expert panel underlined that feedback is essential after thecompletionoftheprocess.

Expertsdiscussedaboutthelocationofcategory45,previouslypartofcriterion9,andfinallyreallocated in criterion 8. They considered that this category is an introductory step to the

61


following criterion “Patient engagement impact”. The experts added the term SMART to thecategory, to define relevance with specific and achievable outcomes. Each letter in SMARTreferstoadifferentcriterionforjudgingobjectives:Specific,Measurable,Assignable,RealisticandTimebound.

In relation to criterion 9, “Patient engagement impact” category 45 was removed from“Patient Engagement Impact” as mentioned before. The rest of the categories were barelychanged,withtheexceptionofsubcategories46.4and46.6.Theyfocusedthecategory46.4inenhancingpatients.Inregardtocategory46.4,thepanelconsideredthat“individualpatients,whoarenotmembersofanyorganisation,shouldbeinvolvedaswell”.

Respectingformersubcategory46.6,itwasrephrasedtoenforceitsmessage.

Finally,categories48and50fromcriterion10,“Sustainability”,wererephrasedtoclarifysometerms.Categories47and49weremerged;asitwasconsideredthatitscombinationgivesmorecontexts and enhances themessage. This criterionwas extensively discussed, because someexperts held that the public entities or patients organisations seek partnerships in order toparticipateinlong-termsustainablepractices,beingnotsoimportantfortheprivatesector.

Oncetheworkonnewspecificationwascompleted,theresultingcategoriesineachcriterionwereweighted. Later, expertsweighed the final 8 criteria by distributing 100 points amongthem.Thehighestweightwasassignedtocurrentcriterion1:“KeyelementsofPracticedesign”(19.5/100),whilethecriteriawiththelowestscorewere“Sustainability”(8/100),followedby“Resources”(10/100).

CurrentcriteriaandcategoryandtheircorrespondingweightsaredetailedinTable9.

Table9:Finalsetofcriteriaandcategories,andtheirweights,recommendedforevaluatingPEinsettingresearchpriorities

Newcriterianames

Criteriaweight

CategoriesCategoryweight

1.KeyelementsofPracticedesign

19.5

Theaimandobjectivesofthepracticeinsettingresearchprioritiesareagreedandunderstandablebyallstakeholders(e.g.patients,academics,researchers,pharmaceuticalcompanies,etc)involvedandrelatedtopatient'sneedsandinterest.

26.0

Thepracticeisbasedonanassessmentofthepatients’needsandpreferences. 22.5

Anyrelevantpolicydirectives,legal,ethics,governancerequirementand/orregulatoryframeworkabouthowtoengagepatientsinsettingresearchprioritieshavebeenconsidered.

11.5

62


Thepatientsengagedinsettingresearchprioritiesreflectdiversityofthetargetpopulation*,patients'circumstances**andvulnerability.*Referstoculturalbackground,social-economicstatus,gender,age,ethnicity,educationallevel,disease,disabilityandvulnerability.**Formerpatients,patientswhoareatriskforadiseasebutdonotyethavethedisease,severityofthedisease,patient’sadvocate,carers,patient’srepresentatives,patientorganizations,levelofexperiences,patient'sscientificknowledge/background).

14.5

Therolesandresponsibilitiesareclearlydefined,agreedandassignedamongallparticipants 12.0

Thepracticeincludesspecifictools*andmechanisms**,toensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.*(e.g.,rolesandresponsibilitycharts,proceduralguidancedocuments,protocols)**(e.g.,meetings,workshops,trainingsessions)

13.5

Totalmustbe100

2.Codeofconduct

11.5

Thepracticeincludesacodeofconduct,whichclearlystatestherulesandprocedures,includingethicalprinciples,ofparticipationinsettingprioritiesforallstakeholders.

44.0

Thetermsandconditionsofconfidentialityagreementsareinplace,presentedinaclearandaccessiblewaytothetargetpopulationinvolved.

30.0

Thepracticecontainsclearandaccessiblemechanismstomanagepotentialconflictsofinterest. 26.0

Totalmustbe100

3.Resources

10.00

Allparticipantsareinformedoftheavailableresources,includingequitablefinancialcompensationframeworktosupportthemduringtheprocess(e.g.,travelandsubsistenceexpenses,timemissedfromwork,child/elderlycareetc.).

27.5

Theinfrastructuresneededforparticipantsareinplace*andadaptedtospecificcircumstancesofparticipants***(e.g.,technologicalsupport,workingspace,communicationtechnologies)**(e.g.possiblephysical,mental,cognitiveoranyimpairment,etc)

41.0

Thepracticeincludesguidance,ready-to-usetoolsandtemplatesmaterialtofacilitateeffectivepatientengagementimplementation. 31.5

Totalmustbe100

4.Capacitybuilding

11.0

Thepracticeidentifiesthecompetenciesthatarerequiredtoperformpatientengagementinsettingresearchprioritiesbyallparticipants. 34.0

Thetrainingmaterialisadapted,comprehensibleandaccessibletoallparticipantstakingintoconsiderationimpairments,literacylevels,culturalbackgroundandthecircumstancesofvulnerablepatientsinvolvedinsettingresearchpriorities.

35.5

Thepracticeensuresthatallstakeholders(otherthanpatients)areadequatelytrainedfortheirrole. 30.5

Totalmustbe100

63


5.Involvementandparticipation

12.0

Patientsreceivetimely,clear,accessibleandappropriateinformationabouttheirinvolvementandwhatisexpectedofthem. 26.0

Thepatient'sparticipationwillbeproperlyplanned,takingintoaccounttimingrequirements,accessibility,andvulnerability. 18.5

Wheneverappropriate,andasagreedbyparticipants,regularfeedbackabouttheoutcomeiscommunicatedinaclearandadaptedway. 20.0

Allparticipantsaregiventheopportunitytogiveregularfeedbackabouttheprocess. 19.0

Thepracticeensuresthereisanamedkeycontactthatpatientscanreachoutthroughouttheprocess. 16.5

Totalmustbe100

6.EvaluationofthePEpracticeinsettingresearchpriority

12.0

Theevaluationframeworkisincludedandsharedamongtheparticipants. 15.5

Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice. 22.0

TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinsettingresearchpriority. 24.5

Theevaluationoutcomesaresharedwithalltheparticipantsandfeedbackisenabledafterthecompletionoftheprocess. 14.5

ThepracticeidentifiesSMART*relevantoutcomesregardingtheinvolvementofpatientsinsettingresearchpriority.*SMART:SpecificMeasurableAssignableRealisticTimebound

23.5

Totalmustbe100

7.Patientengagementimpact

Impacts:

16.0

Researchtopicandprioritiesbecomemoreappropriate,basedonpatients’needs. 22.0

Theproductprofile(drug,technology)becomemorerelevantandusableforpatients. 19.0

Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients. 19.5

Credibilityofsubmittedresearchproposalsareenhancedwheninvolvingpatientsandpatients’organisations. 10.5

Resourceallocationswhenengagingpatientswithinindustrybecomemoreappropriateandarebasedonpatients’needs. 12.5

Completedstudieswillprovidemoreusefulinformationforpatientsanddecision-makingprocesswheninvolvingpatientsandpatientorganisations.

16.5

Totalmustbe100

8.Sustainability

8.00

Thepracticeisembeddedintheinstitutionororganisationand,whenrelevant,theallianceswithprivateand/orpublicinstitutionarefostered.

46.0

Humanandfinancialresourcesareallocatedforthelong-termcontinuityofthepracticeonPEinsettingresearchpriorities. 31.0

64


ThereisadisseminationandcommunicationplandemonstratingtheprocessandoutcomeofPE. 23.0

Totalmustbe100

Totalmustequal100

65


Annex 6. Results of the Delphi on PE in Designing Clinical Trials

ResultsofRound1(Online)–RelevanceassessmentTable10:RelevantcategoriesonPEinDesigningClinicalTrials(Round1)

CriteriaID

Criteria CategoryID

Category

1 Aim andobjectives

3 TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants


7 The patient profiles engaged in the design of clinical trials reflect therelevantdiversityof thepopulationtargeted(e.g.,gender,socioeconomicstatus, ethnicity, cultural background, geographical location etc.) andincludesgendersensitiveinformation.

11 The role and responsibilitiesof allparticipants involved in thedesignofclinicaltrials,includingpatients,areclearlydefinedandagreed.

4 CodeofConduct 15 Thepracticeincludesacodeofconduct,whichclearlystatestherulesandmechanismofparticipationindesignofclinical trials forallstakeholdersinvolved.

16 Effortsaremadetoensurethatallindividualsinvolvedknowtherulesofparticipationinclinicaltrialsdesign(e.g.,workshops,meetings).

17 The terms and conditions of confidentiality agreements are adequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulationinvolved.

18 Potential conflicts of interest are addressed and managed (e.g., policiesthatrequirefulldisclosure,transparency,andaccountability).

19 Thepractices containsmechanisms (e.g., personalizedmeetings,waysofanonymousuncovering)to identifyandaddressdiscriminatory,coercive,intimidation, and unethical behaviours resulting from theirparticipationduringthedesignoftheclinicaltrialsprocess.

20 Funding resources for the practice on PE in design of clinical trials areclearly explained to all stakeholders involved prior to involvement, andanychangesthatoccurduringthepracticearecommunicatedup-front.

5 Resources 21 Allparticipants involvedindesignofclinical trialspracticeare informed,in clear and accessible way, of the resources available to support theirtasksduringtheprocess.

22 An appropriate and equitable financial compensation framework is inplace for all participants (e.g. professionals, patients etc.) invited toparticipate in the PE practice in design of clinical trials (e.g.,reimbursement of expenses for travel, subsistence, time missed fromwork,child/elderlycare).

6 Capacitybuilding 26 Comprehensive and accessible training or induction material andprograms to respond to the need of the people involved in design ofclinicaltrials.


30 Thetypeofinteractionamongthedifferentstakeholders(e.g.,cooperation,co-creation,information,advice…)isclearlydefined.

66


31 Considerationhasbeengiven to specificparticipants’ circumstancesandcharacteristics linked to but not limited to possible physical or mentalimpairments, cultural background, age and other relevant features (e.g.,useoflanguage,formatofmeetings,thevenue,informationprovided).

32 The timing requirement of the engagement in design of clinical trials isappropriately planned, taking into account patient’s needs, and basedupon the requirements of the engagement process (e.g. timeof the day,durationetc.).

35 The information about the outcomes included for each phase of theinvolvementiscommunicatedinplainandrespectfullanguageandusingaccessible, clear and understandable formats, taking into accountpotential impairments(e.g., impairedvision,blindness,anddeafnessandcognitive problems), low literacy levels, cultural backgrounds and otherrelevantfactors(e.g.age,disease/condition,etc.).

36 Asinglepointofcontactfororanamedpersontowhompatientscanreachout to when needed, for information and/or support, throughout theirinvolvementindesignofclinicaltrials.

8 Evaluation 41 The evaluation outcomes are linked to the aim and objectives of the PEpracticeindesignofclinicaltrials.

42 Theevaluationoutcomesaresharedwithall theparticipantsinvolvedinthe practice using appropriate channels and formats suited to theparticipant’scharacteristicsandneeds.

43 The practice includes procedures by which the conclusions of theevaluation are used (e.g., mechanisms to integrate the results of theevaluationinthedesignof thenextpracticeorinthecurrentonewithacontinuousimprovementprocess).

9 PEImpact 45.1 Improvedrecruitmentofstudyparticipants

45.2 Increasedretention(i.e.lowerdropout)ofstudyparticipants

45.4 Betteradherencetotheresearchprotocol,lessamendments

45.6 Faster study completion as a result of improved recruitment andadherencetotheprotocol

45.8 Moreappropriate,inclusive,sensitiveandethicaltrialdesign

10 Sustainability 46 The continuation of the practice is ensured through ownership andinstitutionalororganisationalanchoring.

Table11:NoconsensuscategoriesonPEinDesigningClinicalTrials(Round1)

CriteriaID

Criteria CategoryID

Category

1 Aimandobjectives 1 TheaimandobjectivesofthepracticeonPEindesignofclinicaltrialsare agreed with all the stakeholders (i.e. patients, academics,researchers,pharmaceuticalcompaniesetc.)involved.

2 The aim and objectives of the practice on PE respond to patient’sneedsandinterests.

2 Contextualization of thepractice on PE in the

4 The practice on PE in design of clinical trials is based on a pre-assessmentofpatients’wantsandneedswhenparticipatingindesign

67


DesignofClinicalTrials ofclinicaltrials.

5 A clear understanding of contextual factors that would affect theoutcomes of involving patients in design of clinical trials isdemonstrated (e.g. funding, health system and coverage,characteristicsofthepopulation,socioeconomicenvironment).

6 Any existing policy directives, legal (governance requirements)and/or regulatory framework (e.g. EMA) about how to engagepatientsinclinicaltrials’designhavebeenfollowed.


8 The targetparticipant’sprofilewhoparticipate in thedesignof theclinicaltrialrepresentthetargetpopulationthatwillberecruitedtotheclinical trials(e.g., formerpatients,peoplewhoareat risk for adisease,butdonotyethavethedisease,patients’advocates,patients’representatives,patientorganizations, levelof experience indesignofclinicaltrials).

9 The practice includes methods to identify individuals with theperspectives (or points of view) needed to engage in clinical trialsdesign(e.g.,patient’snetworks,healthcareprofessionals’networks,publicads).

14 The practice includes specific tools (e.g., roles and responsibilitycharts, procedures, procedural guidance documents, protocols) andmechanisms(e.g.,meetings,workshops, training sessions),areusedtoensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.

5 Resources 23 Fundingisallocatedtocoverallelementsofthepractice(e.g.,humanresources,material),andisadequatefortheintendedwork.

24 The adequate infrastructures needed for patients’ and otherstakeholders’ involvement in design of clinical trials practice is inplace (e.g., technological support, working space, information andcommunicationtechnologies)andadaptedtospecificcircumstancesof participants (e.g. possible physical, mental, cognitive or anyimpairmentsetc.)


6 Capacitybuilding 29 Practices on PE in design of clinical trials incorporatetraining/coaching for stakeholders, other than patients, regardinghowtoincorporatepatientsinthedesignofclinicaltrials.


33 Mechanismstoensurethemeaningfulparticipationandcontributionof all participants involved in the PEactivity are in place taking inconsideration characteristics and circumstances of vulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc).

34 A regular feedback system to inform patients about theoutcomes/changesisincludedforeachphaseoftheinvolvement.

8 Evaluation 38 Anevaluationframeworkisincluded(structure,processandresults)anditenablesregularityoftheevaluation(frequency,timing…).

40 Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.

9 PEImpact 44 Thepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinthedesignofclinicaltrials.

68


45.5 Improvedstudyparticipants’trialexperience.

45.9 Moreappropriatewordingand timingof research instruments (e.g.questionnairesandinterventions).

45.10 Improved patient information accessibility (e.g. lay summaries,informationsheets,consentform,recruitmentmaterial).

10 Sustainability 47 Humanandfinancialresourcesrequiredforthelong-termcontinuityof the practice on PE in design of clinical trials are identified andsecured.

48 The practice on PE in design of clinical trials identifies andestablishes alliances with other private and/or public institutions,citizensandothers,toensureitscontinuity,aslongasconfidentialityisrespected.

Table12:DroppedcategoriesonPEinDesigningClinicalTrials(Round1)

CriteriaID

Criteria CategoryID

Category


10 PracticesonPEin in clinical trialsdesign consider the inclusionofotherrelevantpointsofviewseparatetothatofthepatients(e.g.carers, tutors,research and academia, healthcare professional,pharmaceutical/biotechnology/medical technology industry, policymaker,regulator,healthtechnologyassessment(HTA)agency,funder,payer…).

12 There is clearguidanceabouthow toassign roles amongallparticipantsinvolvedininthedesignofclinicaltrials.

13 Thepracticedisplaysflexibilityandopen-mindednessinassigningtherolesforthepatients,accordingtotheprofileandbackgroundofthepatient(e.g.assourcesofinformation,asconsultantsorascollaboratingpartnersoftheprocess).

6 Capacitybuilding 27 The trainingprogramspecifies timewhen trainingneeds tobeprovided,duration and frequency of the sessions (e.g. on demand, once, twice amonth/year).

28 Trainingforpatientsincludesareas,suchaslegalandregulatoryconceptsand challenges in research or deliberative process, concepts (e.g. basicstatisticsetc.),terminologyetc.


37 Mechanismsare in place toguaranteeallmembers have the same statuswithintheworkgroup,regardlessoftheirexperienceandknowledge.


9 PEImpact 45.3 Increaseddiversityofstudyparticipants

45.7 Decreasecostsasaresultofimprovedrecruitment,retentionandpositivetrialexperience

10 Sustainability 49 Thereisadisseminationandcommunicationplan/strategiesoftheresultsin PE in design of clinical trials through culturally appropriate andaccessiblemechanisms.

69


ResultsofRound2(Online)–PrioritizationTable13:RelevantcategoriesonPEindesigningclinicaltrials(Round2)

(Newcategoriesinorange)

CriteriaID

Criteria CategoryID

Category

1 Aimandobjectives2 The aim and objectives of the practice on PE respond to patient’sneedsandinterests.

2 Contextualization ofthepracticeonPE intheDesignofClinicalTrials

6 Any existing policy directives, legal (governance requirements)and/or regulatory framework (e.g. EMA) about how to engagepatientsinclinicaltrials’designhavebeenfollowed.


9 The practice includes methods to identify individuals with theperspectives (or points of view) needed to engage in clinical trialsdesign (e.g., patient’snetworks,health careprofessionals’networks,publicads).

14 The practice includes specific tools (e.g., roles and responsibilitycharts, procedures, procedural guidance documents, protocols) andmechanisms(e.g.,meetings,workshops,trainingsessions),areusedtoensure that all participants understand their own and other’s rolesandresponsibilities.

5 Resources 23 Fundingisallocatedtocoverallelementsofthepractice(e.g.,humanresources,material),andisadequatefortheintendedwork.

24 The adequate infrastructures needed for patients’ and otherstakeholders’ involvement in design of clinical trials practice is inplace (e.g., technological support, working space, information andcommunication technologies) and adapted to specific circumstancesof participants (e.g. possible physical, mental, cognitive or anyimpairmentsetc.)


7 Involvement andParticipation

33 Mechanismstoensurethemeaningfulparticipationandcontributionof all participants involved in the PE activity are in place taking inconsideration characteristics and circumstances of vulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc.).

34 A regular feedback system to inform patients about theoutcomes/changesisincludedforeachphaseoftheinvolvement.

38 Anevaluationframeworkisincluded(structure,processandresults)anditenablesregularityoftheevaluation(frequency,timing…)

8 Evaluation 40 Methods, tools and monitoring system are in place to evaluate PEpractice.

44 Thepractice identifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinthedesignofclinicaltrials.

70


9 PEImpact 45.5 Improvedstudyparticipants’trialexperience

45.9 More appropriatewording and timing of research instruments (e.g.questionnairesandinterventions)

45.10 Improved patient information accessibility (e.g. lay summaries,informationsheets,consentform,recruitmentmaterial)

Newcategory

Identification clinical trial endpoints that better reflect functionaloutcomes/benefitsforpatientsfromusingagiventherapycomparedtotraditional/standardregulatoryendpoints

10 Sustainability 47 Humanandfinancialresourcesrequiredforthelong-termcontinuityof the practice on PE in design of clinical trials are identified andsecured.

Table14:NoconsensuscategoriesonPEindesigningclinicaltrials(Round2)

(Newcategoriesinorange)

CriteriaID

Criteria CategoryID

Category

1 Aim andobjectives

1 TheaimandobjectivesofthepracticeonPEindesignofclinicaltrialsareagreed with all the stakeholders (i.e. patients, academics, researchers,pharmaceuticalcompaniesetc.)involved.

6 CapacityBuilding

29 PracticesonPE in design of clinical trials incorporate training/coachingfor stakeholders, other than patients, regarding how to incorporatepatientsinthedesignofclinicaltrials.

7 InvolvementandParticipation

Newcategory

When engaging patients in the design of clinical trials, patient-orientedoutcomesareobtainedthroughvalidinstruments

Table15:DroppedcategoriesonPEindesigningclinicaltrials(Round2)

CriteriaID

Criteria CategoryID

Category

2 Contextualization ofthe practice on PE inthe Design of ClinicalTrials

4 The practice on PE in design of clinical trials is based on a pre-assessment of patients’ wants and needs when participating indesignofclinicaltrials.

5 A clear understanding of contextual factors that would affect theoutcomes of involving patients in design of clinical trials isdemonstrated (e.g. funding, health system and coverage,characteristicsofthepopulation,socioeconomicenvironment).

71



8 Thetargetparticipant’sprofilewhoparticipateinthedesignoftheclinicaltrialrepresentthetargetpopulationthatwillberecruitedtotheclinicaltrials(e.g.,formerpatients,peoplewhoareatriskforadisease, but do not yet have the disease, patients’ advocates,patients’representatives,patientorganisations, levelofexperienceindesignofclinicaltrials).

10 Sustainability 48 The practice on PE in design of clinical trials identifies andestablishesallianceswith other privateand/orpublic institutions,citizens and others, to ensure its continuity, as long asconfidentialityisrespected.

Table16:RelevantcategoriesonPEindesigningclinicaltrialsandtheirPriority(Round2)

CriteriaID

Criteria CategoryID

Category Priority

1 Aim andobjectives

3 The aim and objectives of the practice on PE are clear andunderstandableforallparticipants

HP

4 CodeofConduct 16 Effortsaremadetoensurethatallindividualsinvolvedknowthe rules of participation in clinical trials design (e.g.,workshops,meetings).

HP

17 The terms and conditions of confidentiality agreements areadequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulationinvolved.

HP

18 Potential conflicts of interest are addressed and managed(e.g., policies that require full disclosure, transparency, andaccountability).

HP

5 Resources 21 Allparticipantsinvolvedindesignofclinicaltrialspracticeareinformed, in clear and accessible way, of the resourcesavailabletosupporttheirtasksduringtheprocess.

HP

22 An appropriate and equitable financial compensationframework is inplace for allparticipants (e.g.professionals,patientsetc.)invitedtoparticipateinthePEpracticeindesignof clinical trials (e.g., reimbursement of expenses for travel,subsistence,timemissedfromwork,child/elderlycare).

HP


31 Consideration has been given to specific participants’circumstancesandcharacteristicslinkedtobutnotlimitedtopossible physical or mental impairments, culturalbackground, age and other relevant features (e.g., use oflanguage, format of meetings, the venue, informationprovided).

HP

35 Theinformationabouttheoutcomesincludedforeachphaseoftheinvolvementiscommunicatedinplainandrespectfullanguage and using accessible, clear and understandableformats, taking into account potential impairments (e.g.,impaired vision, blindness, and deafness and cognitiveproblems),lowliteracylevels,culturalbackgroundsandother

HP

72


relevantfactors(e.g.age,disease/condition,etc.).

8 Evaluation 41 TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeindesignofclinicaltrials.

HP

42 Theevaluationoutcomesaresharedwithalltheparticipantsinvolved in the practice using appropriate channels andformatssuitedtotheparticipant’scharacteristicsandneeds.

HP

9 PEImpact 45.8 Moreappropriate,inclusive,sensitiveandethicaltrialdesign HP

10 Sustainability 46 The continuation of the practice is ensured throughownershipandinstitutionalororganisationalanchoring.

HP


7 The patient profiles engaged in the design of clinical trialsreflect therelevantdiversityof thepopulationtargeted(e.g.,gender,socioeconomicstatus,ethnicity,culturalbackground,geographical location etc.) and includes gender sensitiveinformation.

MP

4 CodeofConduct 15 Thepractice includesacodeofconduct,whichclearlystatestherulesandmechanismofparticipationindesignofclinicaltrialsforallstakeholdersinvolved.

MP

20 FundingresourcesforthepracticeonPEindesignofclinicaltrialsareclearlyexplainedtoallstakeholdersinvolvedpriorto involvement, and any changes that occur during thepracticearecommunicatedup-front.

MP

6 CapacityBuilding

26 Thepractice includes comprehensiveandaccessible trainingorinductionmaterialandprogramstorespondtotheneedofthepeopleinvolvedindesignofclinicaltrials.

MP


30 Thetypeofinteractionamongthedifferentstakeholders(e.g.,cooperation, co-creation, information, advice…) is clearlydefined.

MP

36 A single point of contact for or a named person to whompatients can reach out to when needed, for informationand/or support, throughout their involvement in design ofclinicaltrials.

MP

8 Evaluation 43 Thepracticeincludesproceduresbywhichtheconclusionsofthe evaluation are used (e.g., mechanisms to integrate theresultsoftheevaluation inthedesignof thenextpracticeorinthecurrentonewithacontinuousimprovementprocess).

MP

9 PEImpact 45.1 Improvedrecruitmentofstudyparticipants MP

45.2 Increasedretention(i.e.lowerdropout)ofstudyparticipants MP

45.4 Betteradherencetotheresearchprotocol,lessamendments MP

45.6 Fasterstudycompletionasaresultof improvedrecruitmentandadherencetotheprotocol

MP

73



11 Theroleandresponsibilitiesofallparticipantsinvolvedinthedesignofclinicaltrials,includingpatients,areclearlydefinedandagreed.

LP

4 CodeofConduct 19 The practices contains mechanisms (e.g., personalizedmeetings, ways of anonymous uncovering) to identify andaddressdiscriminatory, coercive, intimidation, andunethicalbehaviours resulting from their participation during thedesignoftheclinicaltrialsprocess.

LP


32 The timing requirement of the engagement in design ofclinical trials is appropriately planned, taking into accountpatient’s needs, and based upon the requirements of theengagementprocess(e.g.timeoftheday,durationetc.).

LP

Round3(Meetinginperson)–DiscussionandFinalPrioritizationDiscussionandPrioritisation


Fromtheinitial10criteriatheexpertsagreedon9criteriacomprisedof32categories,and11subcategories(Table17).

Theexpertsrephrasedcategoriesinordertoavoidredundancyandoverlapping.

Thefollowingparagraphsprovidemoreinformationaboutthekeydiscussionfollowedduringthemeetingofthecategories,whichcreatedmorecontroversyanddebate

Mainly, it was considered that the categories related to patients’ needs and types ofstakeholdersshouldbeclarifiedacrossthedocument.Furthermore,expertspreferredtheterm“patient representatives” to “patients”, as this term includes not only patients but carers,parents/tutors and/or staff from patient organisations too. It was replaced across thequestionnaire.Sometermshavebeenincludedasfootnotes.

Criterion1,“Aimandobjectives”experimentedseveralchanges.Ononehand,categories1and3weremerged,becauseofoverlapping,andcategory2wasrephrasedusingsynonyms.

On the other hand, category 32 was reallocated from “Involvement and participation” asexpertsconsideredthiscategoryshouldbeatthebeginningofthequestionnaire.

Criterion3,“TargetparticipantsinvolvedinPE”containedfourcategoriesattheendofthesecondround,allofthemwererephrasedatthemeeting.

74


Criterion2,“ContextualisationofthepracticeonPEintheDesignofClinicalTrials”)and4, “Code of conduct” were merged into a new criteria named “Legal and ethicalconsideration”.Thesecriteriatriggeredadeepdebate,asthereweredifferentpointsofview.

Allthecategorieswererephrasedwiththeconsensusofthepanel.Besides,expertsreorderedcategories to facilitate the assessment. Language, aims of the categories, recipient of theassessment, etc.were exhaustively discussed.Themerges proposed by the PARADIGM teamwerenotaccepted,asexpertsconsideredthattheconceptscoveredacrossthecriterionwouldbereallydifficulttounderstand.

It isnoteworthy thatwithin this new criterion, category 16 was rephrased with the aim ofmakingclearthatthedescriptionoftheeffortsmadeintheprocessmustbecontemplated.

Likewise, category 18 generated an extensive debate about the management andcommunication of potential conflicts of interest, especially those referred to patients andhealthcareprofessionals.Someexpertsheldthatapatientorapatientrepresentativedoesnotnecessarilyrepresentalltherelevantpointsofview,whichcouldcauseaconflict.Moreover,itshould be clarified, in every practice, what a conflict of interest for a patient/patientrepresentativeis.

It was also pointed out that “…conflicts of interestmust be avoided, notmanaged” in a PEpractice. However, it was discussed that this is not always possible. For example, if theinvolvementofsomehealthcareprofessionalsisavoidedbecauseoftheirconflictsofinterest,experts with strong knowledge in the field would be excluded; therefore, a valuablecontributioncouldbemissed

Likewise, in the discussionof category 20, related to the funding resources for participants,discussionaboutpaymentsandfundingrecipientsweredebated.Therewasdiversityofpointsof views. On one side, some of the experts considered, neither patients nor professionalsshould collect payments for their participation because of the potential conflicts of interest.Otherviewsconsideredthatanyactivityshouldberemuneratedtoanystakeholder,healthcareprofessionals and patients involved because they are contributing with their knowledge,expertise,timeandeffort.

Criterion5,“Resources”,wasminimallychanged.Threeofthecategorieswererephrased.

After the analysis of the results in second-round, it was not clear that category 29 withincriterion6,“Capacitybuilding”,wasrelevant.Finally,itwasrephrased.

Categories 29 and 30 assessed the same activity but in different population, so they weremaintainedseparated.

Criterion7,“Involvementandparticipation”,sufferedseveralchanges.Thepanelofexpertsacceptedthesuggestionofmergingthecategories21and33.Likewise,theyacceptedtomergecategories 34 and 36 into amore comprehensive category, considering that to give regularfeedbackclosestheloopofpatientrepresentatives’involvementintheimplementationofthepractice.

75


Asmentionedbefore,category32wasremovedfromthiscriterionandreallocatedincriterion1.

The new category proposed by an expert in the previous Delphi rounds (Criterion 7:Involvementandparticipation:WhenengagingpatientsintheDesignofclinicaltrials,patient-oriented outcomes are obtained through valid instruments, when presented to the panel ofexperts,itwasdiscardedbecausetheyconsideredthat“validinstruments”weretoodifficulttohaveinthiscontextsotheydiscardedthecategory.

Criterion 8, “Evaluation of the PE practice in the design of clinical trials” was kept asoriginallypresented.

Thepaneldetectedoverlappingamongcategories38and40,allidentifyingdifferentfeaturesforwhichthePEpracticeshouldbeevaluated(methods,toolsandmonitoringsystems).

Categories41and43wererephrased.Inthecaseofcategory42,expertsrephrasedto“closethe feedback loop”. The aim was to “make sure the information is shared and patientrepresentativesreceiveit”.

Category44wasdroppedsinceitwasconsideredtobeconfusing.However,itwasthoughttobetterfitasanewsubcategorywithinthecategory“Impact”.

Criterion9, “Patientengagement impact”, triggeredadeepdebateabouthow tomeasuresomecategories,how tomake the rephrasingethical and inclusive, etc. Finally, categories inthiscriterionwereorganisedassubcategorieswithincategory45:“Impacts”.

Category44wasrephrasedandreallocatedasanewsubcategoryandanewsubcategorywasaddedbythepanel.

Criterion10,“Sustainability”,hastwocategories.Category46wasrephrased;itwassaidthatit’snotalwaysnecessarytoensurethecontinuationofthepracticebecauseitissystematicallyembedded.

Finally,category47waskeptasoriginallypresentedtothepanel.

Oncethisworkwascompleted,theresultingcategoriesineachcriterionwereweightedaswellasthefinal8criteria,bydistributing100pointsamongthem.

The highest weight was assigned to current criterion 1: “Patient engagement impact” and“Aims and objectives” (both of themwith a weight of 14/100), while the criteria with thelowestscorewere“Sustainability”(8/100),followedby“Resources”(10/100)and“EvaluationofthePEpracticeinsettingclinicaltrials”(10/100).

CurrentcriteriaandcategoryandtheircorrespondingweightsaredetailedinTable17.Table17:Finalsetofcriteriaandcategories,andtheirweights,recommendedforassessingPEpracticesinthedesignofclinicaltrials(followingtheassessmentpracticeprocess).

Newcriterianames

Criteriaweight

CategoriesCategoryweight

76


1. Aims andobjectives 14

ThereisgeneralagreementontheaimsandobjectivesofthepracticeonPEandtheseareunderstandablebyallrelevantstakeholdersinvolvedinthedesignofCT.

40.6

Theaimsandobjectivesof thepracticeonPE focusonpatient’sneedsandinterests. 34.7

The timeline for the engagement in the design of clinical trials isappropriately planned and allows for incorporation throughout theprocess.

24.7

Totalmustequal100

2. Targetparticipantsinvolved

12

The patient representatives* engaged in the design of clinical trialsreflect the relevant diversity** of the target population.* Patients, carers, parents or tutors and/or staff from patient’sorganization that act as a liaison between the patients and otherorganisations, represent their interests and advocates for theirrights.** Including but not limited to cultural background, socio-economicstatus, gender, age, ethnicity, educational level, disease, disability,vulnerability,etc.

32.9

ThepracticeofPEincludesacleardescriptionoftheprocessandcriteriafollowedtoidentifypatientrepresentativestoparticipateinthedesignofclinicaltrials.

19.0

The role and responsibilities of all target participants involved in thedesign of clinical trials, including patient representatives, are clearlydefinedandagreed.

23.7

The practice includes a clear description of the process followed* tomake sure thatallparticipantsunderstand theirownandother's rolesand responsibilities.*Includingdocuments,charts,guidance,training,meetings,workshops.

24.4

Totalmustequal100

3. Legal andethicalconsideration

11

The practice includes a code of conduct, which clearly states theprinciplesofparticipationindesignofclinicaltrialsforallstakeholdersinvolved.

18.9

The practice contains procedures to identify and address unethicalbehaviours, towards all stakeholders, during their participation in thedesignoftheclinicaltrialsprocess.

10.9

There is a clear description of efforts made to ensure that allstakeholders involved know the principles of participation in clinicaltrialsdesign(e.g.,workshops,meetings).

11.0

The terms and conditions of all legal agreements are written andcommunicated ina clear andaccessiblewayandadapted to the targetpopulationinvolved.

14.2

Potential conflicts of interest are addressed and managed (up toavoidance). For this, policies that require full disclosure, transparencyandaccountabilityaredeveloped.

15.7

Funding resources for thepracticeonPEin thedesignof clinical trialsareclearlydocumentedandexplainedtoallstakeholdersinvolvedpriorto involvement, and any changes that occur during the practice arecommunicatedup-front.

14.0

77


All relevantpolicydirectives, legal and/or regulatory frameworkhavebeenfollowedwhenengagingpatientsinthedesignofclinicaltrials. 15.3

Totalmustequal100

4.Resources 10

All participants involved in the design of clinical trials practice areinformed, in clear and accessible way, of the resources available tosupporttheirtasksduringtheprocess.

19.3

Aclear,transparentandequitablefinancialcompensationframeworkisinplaceforpatientrepresentativeswhoparticipateinthePEpracticeinthedesignof clinical trials (e.g., reimbursementof expenses for travel,timemissedfromwork,subsistence,child/elderlycare,stipends).

24.1

Fundingisallocatedtocovergovernance,administrationandoperationsof the practice in PE on design of clinical trials, is adequate for theintendedwork.

18.6

Infrastructure is in place to support the practice in PE on design ofclinical trials and is adapted to specific circumstances of patientrepresentatives (e.g. technological support,working space, informationandcommunicationtechnologies).

21.4

Theresourcesincludeguidelines,andready-to-usetoolsandtemplates,materialthatfacilitateseffectivepatientengagement implementationinaclearandaccessibleway.

16.6

Totalmustequal100

5. Capacitybuilding 10

Comprehensive and accessible trainings or induction materials andprograms are available to respond to the needs of the patientrepresentativesinvolvedinthedesignofclinicaltrials.

54.8

PracticesonPEindesignofclinicaltrialsincorporatetraining/coachingfor stakeholders, other than patient representatives, regarding how toincorporatepatientsinthedesignofclinicaltrials.

45.2

Totalmustequal100

6. Involvementandparticipation

11

The type of interaction among the different stakeholders is defined ateachstage(e.g.co-creation,advice,consultation...). 18.4

Therearemechanisms(e.g.language,formatofmeetings,thevenue,timeof the day, etc.) in place to ensure participation of patientrepresentatives, taking into consideration participant’s characteristics,andcircumstancesofvulnerablepopulation.

29.4

Patient representatives receive timelyand regular feedbackabout theoutcomes/changes for each phase of the involvement, in a clear andunderstandableformatandadaptedtopatients'circumstances.

31.7

Aclearpointofcontactisavailabletowhompatientrepresentativescanreachout towhenneeded for informationand/or support, throughouttheirinvolvementinthedesignofclinicaltrials.

20.6

Totalmustequal100

7.Evaluationofthe PE practicein settingclinicaltrials

10

Therearemethods,toolsandmonitoringsystemsinplacetoevaluatethePEpractice. 24.2

Theevaluation criteria are linked to theaimsandobjectivesof thePEpractice. 23.3

78


Theevaluationoutcomesaresharedwithallthestakeholdersinvolvedinthe practice using appropriate channels and formats suited to patientperspectives'circumstancesandneeds.

25.6

The evaluation outcomes are used to improve future PE practices fordesignofclinicaltrials. 26.9

Totalmustequal100

8. Patientengagementimpact

14

Impacts:

Improvedrecruitment. 7.2

Improvedretention(i.e.fewerdropouts)ofstudyparticipants. 6.8

Betteradherencetotheresearchprotocol. 7.1

Feweramendmentstotheresearchprotocol. 7.3

Improvedtrialexperienceforstudyparticipant. 11.1

Moretimelystudycompletion. 7.3More inclusive, sensitive and ethical trial design, which areappropriate for specific needs and circumstances of the targetpopulation.

12.4

Thewordingandtimingofresearch instruments(e.g.questionnairesand interventions) are appropriate for specific needs andcircumstancesofthetargetpopulation.

9.8

Patient information and education materials (e.g. lay summaries,information and education sheets, consent form, recruitmentmaterial)areappropriatetothespecificneedsandcircumstancesofthetargetpopulation.

10.9

Additional study-specific relevant measure are included tocomplementthedescribedones. 5.6

Identificationofmeaningfulendpointsforpatients. 14.5 Totalmustequal100

9.Sustainability 8

Thecontinuationofthepracticeisensuredthroughownershipanditissystematically embedded in the institution or organisation for allstakeholders.

47.8

Humanandfinancialresourcesrequiredforthe long-termcontinuityofthe practice on PE in the design of clinical trials are identified andsecured.

52.2

Totalmustequal100

Totalmustequal100

79


Annex 7. Results of the Delphi on PE in Early Dialogues

ResultsofRound1(Online)–Relevanceassessment

Table18:RelevantcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round1)

CriteriaID

Criteria CategoryID

Category

1 Aimandobjectives 3 The aim and objectives of the practice on PE in early dialogues areclearandunderstandableforallparticipants.

2 Contextualization ofthe practice on PE inEarlyDialogues

6 Any existing policy, legal (governance requirements) and regulatoryframework(e.g.EMA)abouthowtoengagepatientsinearlydialogueswithHTAandregulatorshavebeenfollowed.

4 CodeofConduct 17 Thetermsandconditionsofconfidentialityagreementsareadequate,written in a clear and accessible way and adapted to the targetpopulation.

18 The practice on PE in early dialogues incorporates mechanisms(explainedtoallstakeholdersinaclearandaccessibleway)tomanagepotential conflicts of interest when patients engage with differentstakeholders (e.g., policies that require full disclosure, transparency,andaccountability).

19 Thepracticecontainsprocedures(e.g.,personalizedmeetings,waysofanonymous uncovering) to identify and address discriminatory,coercive, intimidating, and unethical behaviours resulting from theparticipationduringtheearlydialogueprocess.

5 Resources 21 Allparticipants involved inearlydialogueswithHTAand regulatorsareinformedinclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.

22 Thepracticeincludesanequitablefinancialcompensationframeworkfor all participants (e.g. professionals, patients etc.) invited toparticipate in the early dialogues with HTA and regulators (i.ereimbursementofexpensesfortravel,subsistence, timemissedfromwork,child/elderlycareetc.).

23 Funding is allocated to cover all the elements of the practice (e.g.,humanresources,materialresources),andisadequatetotheintendedwork.

24 AdequateinfrastructuresneededforPEinearlydialoguesareinplace(e.g., technological support, working space, information andcommunicationtechnologies)andadaptedtospecificcircumstancesofparticipants (e.g. possible physical mental, cognitive or anyimpairmentsetc.).

25 Thepractice includesguidance,andready-to-usetoolsandtemplatesmaterial that facilitateseffectivepatientengagement implementationinaclearandaccessibleway.

6 Capacitybuilding 26 The practice on PE in early dialogues identifies the competenciesrequired by all stakeholders involved in early dialogues with HTAbodies and regulators (e.g. knowledge and skill such as how to

80


performpatientengagementinearlydialogues,etc.).

28 The practice includes comprehensive and accessible training orinduction material and programs to respond to the need of theparticipantsinvolvedinearlydialogueswithHTAandregulators.


31 Themechanism of interaction among the stakeholder is adapted totheirneeds.(e.g.meetings,online,skype..etc.).

32 Patientsreceivetimely,clear,accessibleandappropriateinformationabouttheirinvolvement(i.e.whatwillbeaddressed,whatisexpectedfromthem,whattheirparticipationwillentail,howthepracticewillbeorganizedetc.)

33 The practice considers specific patients’ circumstances andcharacteristicslinkedtobutnotlimitedtopossiblephysicalormentalimpairments, cultural background, age and other relevant features(e.g., use of language, format of meetings, the venue, informationprovided).

34 The timing of the engagement in early dialogues with HTA andregulators is appropriatelyplanned,basedupon therequirementsofthereviewprocess.

35 The practice on PE in early dialogues sets mechanisms to ensure aproper deliberative process (e.g. meetings, interviews, focus group,workshops…).

36 A regular feedback system to inform patients about theoutcomes/changesisincludedineachphaseoftheinvolvement.

37 The information about the outcomes included for each phase of theinvolvementiscommunicatedinplainandrespectfullanguage,usingaccessible,clearandunderstandableformats,andtakingintoaccountpotentialimpairments(e.g.,impairedvision,blindness,anddeafness),low literacy levels , cultural backgrounds and other relevant factors(e.g.age,disease/condition,etc.).

38 Thepractice includesa singlepointofcontactoranamedperson towhompatientscanreachouttowhenneeded,forinformationand/orsupport,throughouttheirinvolvementinearlydialogues.

8 Evaluation 41 Anevaluationframeworkis included(structure,processandresults)anditenablesregularityevaluation(frequency,timing…).

44 TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinearlydialogues.

45 Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedin the practice using appropriate channels and formats suited toparticipants’characteristicsandneeds.

9 PEImpact 48.1 Betterunderstandingof technologies’ impact in real life context andthequalityoflifeaspects

48.2 Morepracticalandbetterdesigneddevelopmentplanformedicines.

48.3 Higheraccuracyinmeasuringneedsandpreferencesofpatients.

48.4 Better quality of the evidence assessed during the later HTAassessment.

48.5 Moreusefuladvicegiventoaidcompanydecisions.

81


48.6 Higherrelevanceoftheadvicegiventothelocalcontextofapplication

48.10 Inclusion of patients’ relevant outcomes and end-points in clinicaltrialsdatacollectionandanalysis.

Table19:NoconsensuscategoriesonPEinEarlyDialogueswithHTA&Regulators(Round1)

CriteriaID

Criteria CategoryID

Category

1 Aimandobjectives 1 The aim and objectives of PE in Early Dialogues reflects patients’needs,experiencesandexpectationsintheearlydialoguesprocess.

2 Contextualization ofthe practice on PE inEarlyDialogues

4 In thepracticeonPE inearlydialogues, theprocess isbasedon theassessmentofthepatient’spreferencesandneeds.

5 Aclearunderstandingofcontextualfactorsthatinformtheprocessofpatient engagement is demonstrated so that the quality of dialoguewheninvolvingpatientsinearlydialogueswithHTAsandregulatorsisimproved.


7 The profiles of patients engaged in early dialogues with HTA andregulators reflect the relevant diversity of the population targeted.(e.g., gender, socioeconomic status, ethnicity, cultural background,geographicallocationetc.)andincludesgendersensitiveinformation

9 The practice of PE in early dialogues includes methods to identifypatients who match the clinical criteria needed for a particulardialogue(e.g.,patient’snetworks,healthcareprofessionals’networks,publicadvertiseetc.).

10 The practice supports patients to engage and/or reach out tovulnerable groups to involve them in early dialogueswithHTA andregulators,aswellastootherpatients.

11 Relevantpointsofviewotherthanpatients(e.g.carers,parentsetc.)areconsidered.

12 In the practice, the role and responsibilities of all participantsinvolved,includingpatientsareclearlydefinedandagreed.

14 Guidance,trainingadequateisprovidedtoensurethatallparticipantsunderstand their own role and responsibilities, as well as theresponsibilitiesandrolesofothers.

4 CodeofConduct 15 Thepracticeincludesacodeofconduct,whichclearlystatestherulesof participation in early dialogues with HTA and regulators for allparticipantsinvolved.

20 FundingresourcesforthepracticeonPEinearlydialoguesareclearlyexplainedtoallstakeholders involved,priorto involvement,andanychanges that occur during the practice of PE in early dialogues arecommunicatedup-front.

6 Capacitybuilding 27 ThepracticeonPE inearlydialogues includes trainingprogramsonHTA/Regulatory processes/clinical trial design and methods topatients (e.g., written information, in-house training, online,personalizedassistance,andwebinars).

82


30 ThepracticeonPEinearlydialogues incorporatestraining/coachingfor all stakeholders, other than patients, regarding how toincorporatepatientsinearlydialogueswithHTAandregulators.

8 Evaluation 42 The evaluation framework is shared and agreed among theparticipantstofostercollaboration.

43 Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.

46 The practice includes procedures by which the conclusions of theevaluation are used (e.g. mechanisms to integrate the results of theevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess).

9 PEImpact 47 The practice identifies relevant outcomesand impacts regarding theinvolvementofpatientsinearlydialogueswithHTAandregulators.

48.8 TransparencyofHTA/regulationprocesses.

48.11 MutualunderstandingbetweenallinvolvedstakeholdersincludingthepatientcommunityabouttheregulatoryandHTAprocess.

10 Sustainability 50 Humanandfinancialresourcesrequiredforthe long-termcontinuityofthepracticeonPEinearlydialoguesareidentifiedandsecured.

Table20:DroppedcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round1)

CriteriaID

Criteria CategoryID

Category

1 Aim andobjectives

2 The aim and objectives of the practice on PE in early dialogueswere co-designedandagreedwithallthestakeholdersinvolved.


8 PatientsengagedinearlydialogueswithHTAandregulatorsareselectedtorepresent relevant patients’ circumstances (e.g., former patients, patientswho are at risk for a disease, but do not yet have the disease, patients’advocates, patients’ representatives, patient organizations, level ofexperience inearlydialogueswithHTA and regulators,patient’s scientificknowledge/background).

13 There is clear guidance about how to assign roles among all participantsinvolvedinearlydialogues.

4 Codeofconduct 16 ThepracticeonPE inearlydialogues includesmechanisms, adequateandadapted to the target population to ensure that all individuals involvedknowtherulesofparticipationinearlydialogueswithHTAandregulators(e.g.,workshops,meetings).

6 Capacitybuilding 29 The training program specifies time when training needs to be provided,duration and frequency of the sessions (e.g., on demand, once, twice amonth/year).


39 Mechanisms are in place to guarantee all members have the same statuswithintheworkgroup,regardlessoftheirexperienceandknowledge.

40 ThepracticeonPEinearlydialoguesfostersdiscussionandagreementwithpatients about identifying and selecting preferencesabout the ED processwithHTAbodiesandregulators.

9 PEImpact 48.7 Increased public knowledge and awareness of HTA and regulatory

83


processes.

48.9 Acceptabilityofregulatorydecisions/HTAreportsforstakeholders.

10 Sustainability 49 The continuation of the practice is ensured through ownership andinstitutionalorpracticeorganizationanchoring.

51 The practice on PE in early dialogues identifies and establishes allianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensurethecontinuityofthePEinEDaslongasconfidentialityisrespected.

52 There isadisseminationandcommunicationplan/strategiesof theresultsof theprocessofPE inearlydialogues throughculturally appropriateandaccessiblemechanisms.

ResultsofRound2(Online)–PrioritizationTable21:RelevantcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round2)

CriteriaID

Criteria CategoryID

Category


11 Relevantpointsofviewotherthanpatients(e.g.carers,parentsetc.)areconsidered.

12 In the practice, the role and responsibilities of all participants involved,includingpatientsareclearlydefinedandagreed.

14 Guidance, training adequate is provided to ensure that all participantsunderstand their own role and responsibilities, as well as theresponsibilitiesandrolesofothers.

4 Code ofConduct

15 Thepracticeincludesacodeofconduct,whichclearlystatestherulesofparticipation in early dialogues with HTA and regulators for allparticipantsinvolved.

20 Funding resources for the practice on PE in early dialogues are clearlyexplained to all stakeholders involved prior to involvement, and anychanges that occur during the practice of PE in early dialogues arecommunicatedup-front.

9 PEImpact 47 The practice identifies relevant outcomes and impacts regarding theinvolvementofpatientsinearlydialogueswithHTAandregulators.

48.8 TransparencyofHTA/regulationprocesses.

Table22:NoconsensuscategoriesonPEinEarlyDialogueswithHTA&Regulators(Round2)

CriteriaID

Criteria CategoryID

Category

84


1 Aim andobjectives

1 The aim and objectives of the practice respond to patient’s needs whenparticipatingintheearlydialoguesprocess.


9 ThepracticeonPEinearlydialogues includesmethods to identify individualseligible toparticipate inearlydialogueswithHTAand regulators (e.g., agencydatabases, those involved in other parts of HTA or regulatory processes,patients’networks,healthcareprofessionals’networks,publicadvertiseetc.).

10 Thepracticesupportspatientstoengageand/orreachouttovulnerablegroupstoinvolvetheminearlydialogueswithHTAandregulators,aswellastootherpatients.

8 Evaluation 43 Methods,toolsandmonitoringsystemareinplacetoevaluatePEpractice

46 Thepracticeincludesproceduresbywhichtheconclusionsoftheevaluationareused(e.g.mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess).

9 PEImpact 48.11 Mutualunderstandingbetweenall involved stakeholders including thepatientcommunityabouttheregulatoryandHTAprocess.

10 Sustainability 50 Human and financial resources required for the long-term continuity of thepracticeonPEinearlydialoguesareidentifiedandsecured.

Table23:DroppedcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round2)

CriteriaID

Criteria CategoryID

Category

2 Contextualizationof the practice inPE in EarlyDialogues

4 In the practice on PE in early dialogues, the process is based on theassessmentofpatients’preferencesandneeds.

5 A clear understanding of contextual factors that inform the process ofpatientengagementisdemonstratedsothatthequalityofdialoguewheninvolving patients in early dialogues with HTAs and regulators isimproved.


7 The profiles of patients engaged in early dialogues with HTA andregulatorsreflecttherelevantdiversityofthepopulationtargeted.(e.g.,gender, socioeconomic status, ethnicity, cultural background,geographicallocationetc.)andincludesgendersensitiveinformation

6 Capacitybuilding 27 The practice on PE in early dialogues includes training programs onHTA/Regulatoryprocesses/clinicaltrialdesignandmethodstopatients(e.g., written information, in-house training, online, personalizedassistance,andwebinars).

30 ThepracticeonPEinearlydialoguesincorporatestraining/coachingforall stakeholders, other than patients, regarding how to incorporatepatientsinearlydialogueswithHTAandregulators.


85


Table24:RelevantcategoriesonPEinEarlyDialogueswithHTA&RegulatorsandtheirPriority(Round2)

CriteriaID

Criteria CategoryID

Category Priority

1 Aimandobjectives3 The aim and objectives of the practice on PE in earlydialogues are clear and understandable for allparticipants.

HP

4 Codeofconduct 17 Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulation.

HP

18 The practice on PE in early dialogues incorporatesmechanisms (explained to all stakeholders in a clearand accessible way) to manage potential conflicts ofinterest when patients engage with differentstakeholders (e.g., policies that require fulldisclosure,transparency,andaccountability).

HP


36 Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changes is included in each phase of theinvolvement.

HP

37 Theinformationabouttheoutcomesincludedforeachphaseoftheinvolvementiscommunicatedinplainandrespectful language, using accessible, clear andunderstandable formats, and taking into accountpotentialimpairments(e.g.,impairedvision,blindness,and deafness), low literacy levels , culturalbackgrounds and other relevant factors (e.g. age,disease/condition,etc.).

HP

9 PEImpact 48.1 Betterunderstandingoftechnologies’impactinreallifecontextandthequalityoflifeaspects

HP

48.2 More practical and better designed development planformedicines

HP

48.3 Higheraccuracyinmeasuringneedsandpreferencesofpatients

HP

48.10 Inclusionofpatients’relevantoutcomesandend-pointsinclinicaltrialsdatacollectionandanalysis.

HP

2 ContextualizationofthepracticeinPEinEarlyDialogues

6 Any existing policy, legal (governance requirements)and regulatory framework (e.g EMA) about how toengage patients in early dialogues with HTA andregulatorshavebeenfollowed.

MP

4 Codeofconduct 19 The practice contains procedures (e.g., personalizedmeetings,ways of anonymous uncovering) to identifyandaddressdiscriminatory,coercive,intimidating,andunethical behaviours resulting from the participation

MP

86


duringtheearlydialogueprocess.

5 Resources 21 All participants involved in early dialogues with HTAand regulators are informed in clear and accessibleway, of the resources available to support their tasksduringtheprocess.

MP

22 The practice includes an equitable financialcompensation framework for all participants (e.g.professionals,patientsetc.)invitedtoparticipateintheearly dialogues with HTA and regulators (i.ereimbursement of expenses for travel, subsistence,timemissedfromwork,child/elderlycareetc.).

MP

23 Funding is allocated to cover all the elements of thepractice (e.g., human resources, material resources),andisadequatetotheintendedwork.

MP

24 Adequate infrastructures needed for PE in earlydialogues are in place (e.g., technological support,working space, information and communicationtechnologies)andadaptedtospecificcircumstancesofparticipants(e.g.possiblephysicalmental,cognitiveoranyimpairmentsetc.).

MP

25 Thepractice includesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagement implementation in a clear and accessibleway.

MP

6 Capacitybuilding 26 The practice on PE in early dialogues identifies thecompetenciesrequiredbyall stakeholders involved inearly dialogues with HTA bodies and regulators (e.gknowledge and skill such as how to perform patientengagementinearlydialogues,etc).

MP


31 Themechanismofinteractionamongthestakeholderisadaptedtotheirneeds.(e.g.meetings, online,skype..etc).

MP

32 Patients receive timely, clear, accessible andappropriate information about their involvement (i.ewhat will be addressed, what is expected from them,what their participation will entail, how the practicewillbeorganizedetc)

MP

33 Thepracticeconsidersspecificpatients’circumstancesandcharacteristicslinkedtobutnotlimitedtopossiblephysical or mental impairments, cultural background,age and other relevant features (e.g., use of language,formatofmeetings,thevenue,informationprovided).

MP

34 The timingof theengagement in earlydialogueswithHTA and regulators is appropriately planned, basedupontherequirementsofthereviewprocess.

MP

87


38 The practice includes a single point of contact or anamedpersontowhompatientscanreachouttowhenneeded, for information and/or support, throughouttheirinvolvementinearlydialogues.

MP

9 PEImpact 48.4 BetterqualityoftheevidenceassessedduringthelaterHTAassessment.

MP

48.5 Moreusefuladvicegiventoaidcompanydecisions MP

48.6 Higherrelevanceoftheadvicegiventothelocalcontextofapplication.

MP

6 Capacitybuilding 28 The practice includes comprehensive and accessibletrainingorinductionmaterialandprogramstorespondto the need of the participants involved in earlydialogueswithHTAandregulators.

LP


35 ThepracticeonPEinearlydialoguessetsmechanismstoensureaproperdeliberativeprocess(e.g.meetings,interviews,focusgroup,workshops…).

LP

8 Evaluation 41 An evaluation framework is included (structure,process and results) and it enables regularityevaluation(frequency,timing…).

LP

44 The evaluation outcomes are linked to the aim andobjectivesofthePEpracticeinearlydialogues.

LP

45 The evaluation outcomes are shared with all theparticipantsinvolvedinthepracticeusingappropriatechannels and formats suited to participants’characteristicsandneeds.

LP


DiscussionandFinalPrioritisation


Fromtheinitial10thematiccriteria,theexpertsagreedon8criteriamadeupof29categoriesand7subcategories(seetable25).

Foreachcriterionandcategory,consensuswasreachedonwhethertoretain,rephrase,mergewithanothercategory,ordiscardit.

Thefollowingparagraphsprovidemoreinformationaboutthekeydiscussionfollowedduring

88


themeetingofthecategorieswhichcreatedmorecontroversyanddebate

In general, it was agreed by the expert panel to use the term "PE practice" throughout thediscussion.Furthermore,theterm"patient"wasalsoreplacedwith"patientrepresentative",inordertoreflectthatthisisarolethatrepresentsthewholetargetpopulation.Expertsagreedonremovingbracketsalongthedocumenttobereadwithease.

Criterion1,“Aimandobjectives”consistedoftwocategories.Category1wasbarelymodified.

Criterion 2, “Contextualization of the practice on PE in Early Dialogues” was dropped,movingCategory6toCriterion4.

Criterion 3, “Target participants involved in patient engagement” consisted of fivecategories,twoofwhichweremerged.Thiscriteriongenerateddiscussiononconceptsrelatedto representativeness and diversity of the participants involved in ED. They flagged thedifficultytofindapatientwiththerightprofileorspecificexperienceforaparticularED,andwith the capacity to participate. They also acknowledged that the problem of targetingindividual patients led to end up contacting the same patients due to the difficulty itencounterstofindtherightpatient.

Category 9 was slightly modified and experts decided to add the concept of “vulnerablepopulation” to the glossary (see Annex 4). Category 10was rephrased to reflect that, whenassessing the practice, evaluators should take into account whether the practice tries tocapture diversity (regardless ofwhether it ultimately succeeds). Likewise, Category 11wasmodified toshow therelevanceof representingotherpointsof view,different from thoseoftheirspecificpatients.Finally,Category12and14weremerged;theexpertpanelagreedthatallparticipants inaPEpracticeshouldbeclearabout theirrole.Toachievethis, informationandguidanceshouldbeprovided.

Criterion 4, “Code of conduct” contained four categories. All of themwere rephrased andCategory6was included,movedfromCriterion2“Contextualizationof thepracticeonPE inEarlyDialogues”.Categories15,17and19wererewrittentosimplifyandavoidoverlappingconceptsandCategory18was rephrased toreflect everykindof conflictsof interest.For itspart, Category 6 was modified by including concepts about the regulatory frameworkdeterminedbybodiessuchastheEMA(European,national,etc.).

Criterion5,“Resources” included6categories,oneofthemwasdiscardedandtherestwererephrased.Firstly,Category20wasrephrasedtoincludethatfinancialarrangementrepresentstheframeworkestablishedbythedifferentpartiesintheED.Ontheotherhand,Category21was dropped due to its contentswere also consideredwithin Category20. Category 22wasrewritteninordertoclarifythedefinitionoffinancialframework.Thus,explicitmentionwasmadetotheideathatPEpracticesaresubjecttoacontext/rules.Besides,Categories23and24were rephrased due tooverlaps and to clarify concepts. ThenCategory 25was rewritten toindicatethattoolsusedshouldbebasedonexistingmethodology.

Criterion 6, “Capacity building” was renamed "Capability building". The two existing

89


categories in this criterion were maintained, one of themwas rephrased. Category 26 wasmodifiedinordertodifferentiatepatients’experiencefromtheknowledgerequirements.

Criterion 7, “Involvement and participation” initially consisted of six categories, one ofwhichwasdiscardedandtwoweremerged,theremainingcategorieswererephrased.HenceCategory31wasmergedwithCategory33andCategory32wasdiscardedsincetheroleofthepatient representative was already described. Category 34 was adapted to explain that theprocessplanshouldtakeintoaccountPEpreparationtime,withaviewtoinvolvethepatientfrom the beginning and Category 35was rewritten to determine that an equal deliberativeprocessisrequired.Likewise,Category36wasrephrased.Expertsplacedemphasisonhavingan iterative processwhen including patients in ED. It was considered important to providefeedbackabout their contributionduring theEDprocess. However, in thediscussion itwaspointedoutthedifficultyofgivingfeedbackwithoutbreakingconfidentiality,duetothenatureofthisinformation.Itwasalsohighlightedthatpatientsshouldbeabletoprovidefeedbackontheinformationreceivedrelatedtoresults,documents,finaloutput,etc.Foritspart,Category37 was amended to highlighted the importance of considering possible "disabilities andimpairments"ratherthanlimitationsduetoculturalbackgroundorlowliteracy.Category38was adapted to take into account that contact information should be updated, because ofchangesthatmayoccurwithintheinstitutions.

Criterion 8, “Evaluation of the PE practice in early dialogues with HTA bodies andregulators”was composed of five categories.An additional categorywas included from the"Patient Engagement Impact" criterion, three categories were merged and the rest wererephrased.Category47wasaddedasananalogouscategorytothatexistingwithinthe"PatientEngagement Impact" domain; in this case the word "impact" was removed to avoidoverlapping.Likewise,Categories41,43and44weremergedbasedontheproposalmadebythe PARADIGM Delphi Team. Furthermore, Categories 45 and 46 were rephrased to avoidredundancies.

Criterion9, “PatientEngagement Impact” consistedof twomain categories,Categories47and 48, which experts decided to merge and rephrased. The panel discussed about thepracticaldifficultiesforobservationandmeasurementthemwhenthePEpracticeisassessed.ThereforesubcategorieswereinterpretedjustasalistofexamplesofpotentialimpactsofaPEpracticeonED,accordinglythepanelagreednotassigningweightstoeachsubcategory.

Individually,thesubcategorieswerediscussedbythepanel:Subcategories48.1and48.2wererephrased.BesidesSubcategory48.3andSubcategory48.4wererephrased;andSubcategory48.5 was discarded because overlaps with Category 48. Subcategory 48.8 was adapted todescribetheconceptoftransparencyandSubcategory48.10remainsunchanged,butitsorderwasmodifiedtoimprovethecoherenceofthelistofexamplesofpotentialimpacts.Eventually,Subcategory48.11isdiscardedbecauseofitsoverlapwithSubcategory48.8.

Criterion10,“Sustainability”itwasproposedtomergewithCriterion"Resources",giventhatissuesrelatedtosustainabilityhadbeencovered.Finally,itwasestablishedbythepanelthatthiscriterionanditscontentscouldbeincludedsomewhereelseinthePARADIGMproject,but

90


itshouldnotbeanelementtobeevaluatedineachPEpractice.

Oncetheworkonspecificationwascompleted,theresultingcategoriesineachcriterionwereweighted.Later, expertsweighed the final8 criteriabydistributing100pointsamong them.ThehighestweightwasassignedtocurrentCriterion1,“Aimandobjectives”(17.9/100),whilethecriteriawiththelowestscorewere“Evaluation”(8/100),followedby“PatientEngagementImpact”(10/100).

Currentcriteriaandcategoryandtheircorrespondingweightsaredetailedintable25.Table25:Finalsetofcriteriaandcategories,andtheirweights,recommendedforassessingPEpracticesinearlydialogues

CriteriaCriteriaweight

N Categories Weight

1.Aimandobjectives 17.9

1 The practice aims to meet patients' expectations whenparticipatingintheearlydialogueprocess. 46.5

2 TheaimandobjectivesofPEinEarlyDialoguesareclearandunderstandableforallstakeholdersinvolved. 53.5

Totalmustbe100

2.Targetparticipantsinvolvedinpatient

engagement

15.3

3

The practice of PE in early dialogues includes rigorousmethodstoidentifypatientsorpatientrepresentatives*fromthe relevant target population needed for a particulardialogue.*Patients, vulnerable population, carers, parents or tutorsand/orstaff frompatient’sorganization thatactasa liaisonbetween the patients and other organizations, representtheirinterestsandadvocatesfortheirrights.

30.5

4 The process captures the diversity of the target populationandtheirrangeofperspectives. 25

5 Relevantpointsofviewotherthanthosepatients(e.g.carers,parentsetc.)arealsoconsidered. 16.5

6Guidanceandadequatetrainingareprovidedtoensurethattheroleandresponsibilitiesofallstakeholdersinvolvedareclearlydefinedandunderstoodbyall.

28

Totalmustbe100

3.Codeofconduct

11.3

7

Any applicable policies, legislative and regulatoryframeworks (e.g. EMA) about when and how to engagepatients in early dialogues with HTA and regulators havebeenfollowed

26.5

8Thepracticeincludesacodeofconduct,whichclearlystatesthe rules of participation in early dialogues for allstakeholdersinvolved.

25

9 The terms and conditions of all policies and confidentiality 16.5

91


agreementsareadequate,clearandaccessible.

10 The PE practice incorporates mechanisms to disclose andmanageallpotentialconflictsofinterestofallstakeholders. 18

11

ThePEpracticecontainsprocedurestoidentifyandaddresspotential discriminatory, coercive, intimidating, andunethical behaviours before, during and after theparticipation.

14

Totalmustbe100

4.Resources 10.9

12 Thepractice uses relevant PEmethodologies, guidance andtools. 20.2

13

The practice includes a fair financial framework* forparticipatingpatientrepresentatives.

*Mechanismforpayment/reimbursement

19.7

14 Sufficientfundingisinplacetoensurethatalltheelementsofthepracticearecoveredinitsentirely. 19.7

15

Financial arrangements for the practice of PE in earlydialoguesareclearlyexplained toall stakeholders involved,priortoinvolvement,andanychangesthatoccurduringthepracticearecommunicatedup-front.

17.2

16

An adequate infrastructure is in place (e.g., technologicalsupport, working space, information and communicationtechnologies) and adapted to specific requirements ofparticipants (e.g. possible physicalmental, cognitive or anyimpairmentsetc.).

23.2

Totalmustbe100

5.Capacitybuilding 11.1

17The practice on PE describes the relevant competencies,expertise and experience required by all stakeholders toeffectivelyengageinthisprocess.

42.9

18

ThepracticeincludescomprehensiveandaccessibletrainingorinductionmaterialandprogramstorespondtotheneedsoftheparticipantsinvolvedinearlydialogueswithHTAandregulators.

57.1

Totalmustbe100

6.Involvement

andparticipation

14.7

19Theprocessrespectstheneedforappropriateplanningandpreparation time, to allow patient representatives toeffectivelyengagefromthebeginning.

19

20

Themechanismofinteractionwithpatients’representativesis adapted to their needs to ensure effective PE. Morespecifically, itconsidersspecificpatients’circumstancesandcharacteristics linked tobutnot limited topossiblephysicalormental impairments, cultural background, age and otherrelevant features (e.g., recordings, virtual communication,

28.5

92


use of language, format of meetings, the venue, andinformationprovided).

21The practice on PE in early dialogues sets mechanisms toensure a fair deliberative process* that allows equalopportunityforparticipants'contribution.

16.2

22

A regular feedback system is in place to inform patientrepresentativesabouttheoutcomes/changes,asappropriate,includingtheoptionforpatients’representativestocommentonthefinaloutput.

10.8

23

Theinformationabouttheoutcomesincludedforeachphaseof the involvement is communicated in clear and plainlanguage, using accessible formats, and taking into accountpotentialdisabilitiesandimpairments,asappropriate.

10

24

Thepracticeincludesanup-to-datesinglepointofcontactora named person to whom patients can reach out to whenneeded, for information and/or support, throughout theirinvolvementinearlydialogues.

15.5

Totalmustbe100

7.Evaluation 8.8

25

Thepracticeincludesanevaluationframeworklinkedtotheaimandobjectives, includingmethods,toolsandmonitoringsystem,whichenables systematicevaluation atappropriatephasesoftheprocess.

29.8

26The practice identifies relevant outcomes regarding theinvolvement of patients in early dialogues with HTA andregulators.

31.8

27 Theevaluationoutcomesaresharedwithalltheparticipantsinvolved. 16.6

28Thepracticeincludesproceduresbywhichtheconclusionsofthe evaluation are used to support a continuousimprovementprocess.

21.8

Totalmustbe100

8.PatientEngagementImpact

10 29

Thepracticeidentifiespotentialimpactfromthepractice,forexample:

● Better understanding of the impact of healthtechnologiesand treatments in real life contextand thequalityoflifeaspects.

● Morepracticalandbetterdesigneddevelopmentplanforhealthtechnologiesandtreatments.

● Betterreflectionoftheneedsandpreferencesofpatientsindevelopmentplansfornewtechnologies.

● Inclusionofpatients’ relevantoutcomesandend-points

93


inclinicaltrialsdatacollectionandanalysis.

● Better quality of theevidenceassessedduring the laterregulatoryandHTAassessment.

● Higherrelevanceoftheadvicegiventothelocalcontextofapplication.

● Transparencyof regulatoryandHTAprocesses leads tobetter understanding and trust of the scientificprocedures.

d1.2 criteria to assess how patient engagement ... · d1.2 criteria to assess how patient...

Documents