Embracing collaborative D & R in eHealth

Claudia Pagliari PhD

MedNet, Toronto, Oct 2006

Senior Lecturer in Primary CareChair Edinburgh eHealth Research Network

Division of Community Health Sciences University of Edinburgh

AimsAims

Highlight issues, benefits/barriers for Highlight issues, benefits/barriers for interdisciplinary working in health informaticsinterdisciplinary working in health informatics

Focus on potential for collaboration between Focus on potential for collaboration between software developerssoftware developers & & health services researchershealth services researchers

Consider implications for effective product design & Consider implications for effective product design & implementationimplementation

Illustrate common ground in concepts & Illustrate common ground in concepts & methodologies methodologies

Suggest ways to increase transdisciplinaritySuggest ways to increase transdisciplinarity

The problemThe problem Expert analyses of potential synergies between Expert analyses of potential synergies between

HSR and medical informatics existHSR and medical informatics exist

However their impact on eHealth developments, However their impact on eHealth developments, outside the context of academic studies, is outside the context of academic studies, is questionablequestionable

In practice most eHealth software developments, In practice most eHealth software developments, and the HSR projects associated with them, take and the HSR projects associated with them, take place outside the specialist “Medical Informatics” place outside the specialist “Medical Informatics” community community

As a result developers and HSRs regularly work in As a result developers and HSRs regularly work in parallel universes, each regarding the other’s parallel universes, each regarding the other’s domain of activity as separate and neglecting the domain of activity as separate and neglecting the potential for useful interactionpotential for useful interaction

Out of the basement:Out of the basement:Changing stakeholders in Changing stakeholders in

eHealtheHealth

Clinical needsPolicy needs

Research drivers (e.g. artificial intelligence,

decision science, epidemiology)

Time Stakeholders Technical Focus Objectives Disciplinary drivers Breadth & Complexity

Health system managersIT experts

Health care organisations

InfrastructuresHardware

Operating Systems Databases

Managing clinical & organisational information (e.g. billing, purchasing)

Specialist(e.g. computer science)

Medical researchersHealth professionals

Support staff

Clinical administration systems

Clinical decision support systems

Auditing practiceImplementing procedures

PolicymakersSystem vendors

Patients

Medical imagingElectronic libraries

e-prescribing, booking etc.Clinical eMail

Supporting evidence-based practice

Changing professional behaviour

Engaging cliniciansEngaging patients & the

public

[Medicine, ethics, law,

economics, business, sociology, social anthropology, psychology,

information science, policy studies,

bio-informatics, e-science

General PublicMass Media

Internet based health information/support

Electronic consultingMobile disease monitoring

Supporting health self-management

Protecting patient confidentiality and safety

Interdisciplinary

Complexity & interdisciplinaryComplexity & interdisciplinary

Increasing heterogeneity of topics, research Increasing heterogeneity of topics, research questions and stakeholdersquestions and stakeholders

++

blurring of boundaries between scientific, blurring of boundaries between scientific, policy & commercial R&Dpolicy & commercial R&D

generate challenges for interdisciplinary generate challenges for interdisciplinary working & translation of research to practice working & translation of research to practice

These include the management of non-shared These include the management of non-shared concepts and languages and the value ascribed concepts and languages and the value ascribed to different forms of scientific & technical to different forms of scientific & technical endeavourendeavour

The need for more ‘R’ in ‘D’The need for more ‘R’ in ‘D’ Growing emphasis in medical informatics on Growing emphasis in medical informatics on

resolving ‘human & organisational’ barriers to resolving ‘human & organisational’ barriers to implementationimplementation

2 key themes2 key themes Clinical appropriateness & usability has been Clinical appropriateness & usability has been

compromised by insufficient end-user compromised by insufficient end-user engagement in the design process engagement in the design process

Effectiveness of emerging eHealth Effectiveness of emerging eHealth technologies is unproven, reducing clinicians technologies is unproven, reducing clinicians willingness to use systems willingness to use systems

Joint thinking between developers & Joint thinking between developers & researchers needed to address theseresearchers needed to address these

Barriers to mutualityBarriers to mutuality

Few developers or health service evaluators are Few developers or health service evaluators are ‘medical informaticians’ with interdisciplinary ‘medical informaticians’ with interdisciplinary trainingtraining

Individual eHealth projects may be only part of Individual eHealth projects may be only part of a wider portfolio of D&R, restricting willingness a wider portfolio of D&R, restricting willingness to invest in learning others’ methods and to invest in learning others’ methods and modus operandi modus operandi

Lack of mutual understanding creates a Lack of mutual understanding creates a perception of ‘them’ and ‘us’, which contributes perception of ‘them’ and ‘us’, which contributes to silo or at best parallel workingto silo or at best parallel working

The capacity for key ideas & methods from The capacity for key ideas & methods from each field to translate and inform the other and each field to translate and inform the other and the wider process is therefore limitedthe wider process is therefore limited

Scientific subculturesScientific subcultures

The ‘D’ world of software The ‘D’ world of software designdesign

Application of computer scienceApplication of computer science Roots in engineering & maths Roots in engineering & maths (although draws (although draws

on philosophy & social science e.g. in HCI research)on philosophy & social science e.g. in HCI research) Focus – building machines & softwareFocus – building machines & software Strong links with the business sector Strong links with the business sector Economic drivers prioritise production over Economic drivers prioritise production over

evaluationevaluation Rapid application development with small Rapid application development with small

convenience samples commonconvenience samples common No incentive to engage with researchers No incentive to engage with researchers

The ‘R’ world of HSRThe ‘R’ world of HSR MultidisciplinaryMultidisciplinary Focus on generating evidence to guide clinical Focus on generating evidence to guide clinical

practice & policy (effectiveness, efficiency)practice & policy (effectiveness, efficiency) Largely driven by service & policy needsLargely driven by service & policy needs Uses traditionally ‘medical’ approaches (e.g. Uses traditionally ‘medical’ approaches (e.g.

epidemiology) alongside social & economic epidemiology) alongside social & economic methodsmethods

Strong emphasis on robust and replicable Strong emphasis on robust and replicable research methods (qualitative + quantitative) research methods (qualitative + quantitative)

Prioritises demonstrations of effectiveness using Prioritises demonstrations of effectiveness using hard data, but also explores subjective responseshard data, but also explores subjective responses

so…so…

Software design is mainly concerned Software design is mainly concerned with developing interventions and HSR with developing interventions and HSR with evaluating them with evaluating them

BUT the reality not so clear-cut… BUT the reality not so clear-cut…

Much of HSR aims to inform the design Much of HSR aims to inform the design of new interventions, whilst quality of new interventions, whilst quality software development encompasses software development encompasses multiple evaluation processes multiple evaluation processes

Compatibilities in evaluation Compatibilities in evaluation and design across disciplinesand design across disciplines

Stage-based development, coupled with continuous quality appraisal

Involvement of users

Overlapping methodological toolkits

Key differences relate to methodological standards (sampling, data analysis) + levels of evidence sought

Lifecycle models in software Lifecycle models in software designdesign

Evaluation

Conceptual/formal design

RequirementsspecificationPrototyping

task/functionalanalysis

Implementation

Waterfall Spiral

Star

http://www.levela.com/software_life_cycles_swdoc.htmhttp://www.levela.com/software_life_cycles_swdoc.htm

ISO Standard for human-ISO Standard for human-centred design of interactive centred design of interactive

systemssystems

Sequential stages in Sequential stages in evaluation of complex evaluation of complex

healthcare interventionshealthcare interventions

(Also used in evaluation of new drugs, from initial pre-clinical (Also used in evaluation of new drugs, from initial pre-clinical research through to post-marketing surveillance)research through to post-marketing surveillance)

Iterative view of complex Iterative view of complex intervention evaluationintervention evaluation

(Recognises that results from individual phases may prompt revisions & repetition)

Action Research Action Research SpiralSpiral

Plan-Do-Act Cycle Plan-Do-Act Cycle of Total Quality of Total Quality Management Management

Hybrid Hybrid modelmodel

Evaluation of concepts & prototypes

Maturity: ideas, mock-ups, prototypes

Settings: lab, user spaces)

Define/redefine intended function (e.g. self-management

support) and specific requirements (things the system needs to do, standards it should

meet)

Test prototype in laboratory or in small field trials to assess: functionality, usability, accessibility, perceived utility & acceptability. Formative studies may also assess outcomes such as knowledge or task processing time.

Develop/refine prototype to meet stated aims

Assess impact Hard: clinical outcomes, cost Soft: e.g. perceived value or

barriers, quality of life Process: e.g. uptake

Unintended consequencesPragmaticEvaluation

Maturity: Product shown to be effective, usable & acceptable in lab & field

environmentsSettings: As rolled out

Experimental studies (e.g. RCTs) Quasi experimental studies (e.g.

CBA) Qualitative studies (e.g. exploring

perceptions & attitudes)Mixed methods approaches

RigorousEvaluation

Maturity: pre-tested, iterated, user-

informed versionsSettings: Sample

settings in which the resource will be implemented.

Ongoing monitoring of uptake, effectiveness and safety in routine

practice

Implement in routine clinical

practice

Refine/repeat if indicated by the results of above

Refine if indicated. (Changes may occur during the trial in response to problems identified, or at the end.)

Generate ideas/concepts/ theory

Overlapping empirical Overlapping empirical techniquestechniques

Software & Usability EngineeringSoftware & Usability Engineering

Inquiry Inquiry (assessing needs, relevance, (assessing needs, relevance, possible barriers)possible barriers)

Contextual inquiry (requirements Contextual inquiry (requirements gathering, assessment of prototypes in gathering, assessment of prototypes in intended setting); Ethnographic study/Field intended setting); Ethnographic study/Field Observations; Surveys; Questionnaires; Observations; Surveys; Questionnaires; Journaled Sessions; Self reporting logs; Journaled Sessions; Self reporting logs; Remote usage observation; Screen Remote usage observation; Screen snapshotssnapshots

Inspection Inspection (during (during developmentdevelopment)) Heuristic evaluation; Cognitive Heuristic evaluation; Cognitive

walkthroughs; Formal Usability Inspection; walkthroughs; Formal Usability Inspection; Pluralistic walkthroughs; Feature Pluralistic walkthroughs; Feature Inspection; Consistency Inspection; Inspection; Consistency Inspection; Standards Inspection; Guideline checklists; Standards Inspection; Guideline checklists; Thinking aloud protocol; PrototypingThinking aloud protocol; Prototyping

Testing (formative)Testing (formative) Thinking aloud protocol; Co-discovery Thinking aloud protocol; Co-discovery

methods; Question asking protocol ; methods; Question asking protocol ; Performance measurement ; gaze trackingPerformance measurement ; gaze tracking

Related techniquesRelated techniques Action research Blind voting; Card sorting; Action research Blind voting; Card sorting;

Archetypal researchArchetypal research

Health Services ResearchHealth Services Research

Inquiry Inquiry Needs assessmentNeeds assessment; Ethnographic studies; ; Ethnographic studies;

Participant and non-participant observation; Participant and non-participant observation; Face-to-face interviews; Document analysis; Face-to-face interviews; Document analysis; Telephone surveys; Postal questionnaires; Telephone surveys; Postal questionnaires; Focus groups; Focus groups; InterviewsInterviews

Inspection Inspection (during the evaluation (during the evaluation process)process)

Observational studies (epidemiological, Observational studies (epidemiological, records based); Interviewsrecords based); Interviews

Testing (summative)Testing (summative) Impact assessment (Randomised Impact assessment (Randomised

controlled trial; Controlled before & after controlled trial; Controlled before & after study; Interrupted time series; Case study; Interrupted time series; Case control study; Cost-benefit analysis; Other control study; Cost-benefit analysis; Other rigorous designs); Qualitative outcomes rigorous designs); Qualitative outcomes assessment (rigorous qualitative data assessment (rigorous qualitative data analysis using sociological methods)analysis using sociological methods)

Related techniques Related techniques Action research/Participative evaluation; Action research/Participative evaluation;

Continuous quality improvement; Conjoint Continuous quality improvement; Conjoint analysisanalysis

Overlap but different primary goals Overlap but different primary goals of evaluationof evaluation Quality productQuality product Evidence of effectivenessEvidence of effectiveness

Depth/rigor of techniques variesDepth/rigor of techniques varies

Benefits of collaborationBenefits of collaboration Strengthen quality of evaluations and enhance Strengthen quality of evaluations and enhance

evidence base, thus facilitating better policy and evidence base, thus facilitating better policy and purchasing decisions. purchasing decisions.

Accrue economic benefits by demonstrating that Accrue economic benefits by demonstrating that systems are effective, cost effective & safe, as systems are effective, cost effective & safe, as well as highly robust, accessible, acceptable & well as highly robust, accessible, acceptable & usable usable

Opportunity for expert evaluators to influence the Opportunity for expert evaluators to influence the development of interventions prior to clinical trialsdevelopment of interventions prior to clinical trials

May help HSRs recognise cognitive barriers to ICT May help HSRs recognise cognitive barriers to ICT adoption hence providing context for adoption hence providing context for interpretation of large study results interpretation of large study results

Adopting rigorous and replicable methods Adopting rigorous and replicable methods facilitates publication & disseminationfacilitates publication & dissemination

Challenges for Challenges for interdisciplinarityinterdisciplinarity

Understanding each others’ theoretical roots, methods, external Understanding each others’ theoretical roots, methods, external drivers & techniques drivers & techniques

Developing transdisciplinary experts, familiar with both skill setsDeveloping transdisciplinary experts, familiar with both skill sets Balancing quest for collaboration against risk of compromising Balancing quest for collaboration against risk of compromising

disciplinary strengths disciplinary strengths Balancing need for innovation and rapid delivery, with pressure Balancing need for innovation and rapid delivery, with pressure

to adopt methodologically robust standards to adopt methodologically robust standards Establishing trust and mutual respect Establishing trust and mutual respect Developing sufficient understanding of each discipline’s portfolio, Developing sufficient understanding of each discipline’s portfolio,

to be able to tailor methods appropriately to R&D problemsto be able to tailor methods appropriately to R&D problems Educating commissioners, who have traditionally held different Educating commissioners, who have traditionally held different

expectations for design and evaluation projects re. expected expectations for design and evaluation projects re. expected outputs (e.g. new products vs. new knowledge) & methodologies outputs (e.g. new products vs. new knowledge) & methodologies (e.g. user-centred design vs. studies of clinical impact)(e.g. user-centred design vs. studies of clinical impact)

EndEnd

Contact:Contact:

claudia.pagliari@ed.ac.ukclaudia.pagliari@ed.ac.uk