evaluation methods in heathcare systems

1

EVALUATION METHODS IN HEALTHCARE SYSTEMS

3

Introduction: From 1995s in the hope of :

Medical information systems involve:

Computer-stored databases• patient

information to support medical order entry

• Results reporting

• Decision support systems

• clinical reminders

Comprehensive systems

• coordinates patient care activities by linking computer terminals in patient care areas to all departments

Smaller separate systems

•Link patient care areas to only one department

•Laboratory system•Radiology system•Pharmacy system•Expert systems•Computerized databases

CPOE

• Concerns about patient safety and medical error

• Need to Evaluation

Unfortunately reports of system failures have continued

1.Anderson, J.G. and C. Aydin, Evaluating the Organizational Impact of Healthcare Information Systems. 2005: Springer.pp 5-8.

Hope• Increasing efficiency, reducing costs, and improving patient care

1995 • Healthcare applications appeared

4

Evaluation Definition: Evaluation can be defined as “the act of measuring or exploring properties

of a health information system (in planning, development , implementation, or operation),the result of which informs a decision to be made concerning that system in a specific context.”

Evaluating should perform not only in technology assessment but also in the social and behavioral processes

1.Ammenwerth, E., et al., Evaluation of health information systems—problems and challenges. International Journal of Medical Informatics, 2003. 71(2-3): p. 125-135.2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

During IT Evaluation

Not only the technology itself

But also the interaction between ITand human players in their information processing

Must be Considered

5

Evaluation is the means to assess:

5

Quality

Value

Impacts of IT in the health care

environment

Effects

6

Aim of evaluation:

To provide the basis for a decision about the IT system investigated

Decision-making context is also the context of the evaluation .[2]

1- Yusof, M. M., et al. (2008). "Investigating evaluation frameworks for health information systems." Int J Med Inform 77(6): 377-385.

2-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Press.p 9-11.

Improve performance

Outcomes

Safety

Effectiveness

7

Questions should be answered: Evaluation seeks to answer the :

Why (objective of evaluation), Who (which stakeholders’ perspective is going to be evaluated) When (which phase in the system development life cycle), What (aspects or focus of evaluation)(What is it going to be used for ?) How (methods of evaluation) questions [1],[2]

1- Maryati Mohd. Yusof, R.J.P., Lampros K. Stergioulas. Towards a Framework for Health Information Systems Evaluation. in Proceedings of the 39th Hawaii International Conference on System Sciences. 2006. Hawaii IEEE.2-SYMONS, V.J., A review of information systems evaluation-content ,context and process. Eur J Inf Syst, 1991. 1(3): p. 205-212.

8

1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2 006: Springer .p 6.

9

SDLC

Maintenance

Implementation

Testing

Design

Analysis

Planning

10

Pre-implementation evaluation

Inform difficult decisions

Prior to starting any program

As well as post-implementation evaluation

1.Nykanen, P., et al., Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform, 2011. 80(12): p. 815-27.

11

Assessment:There are two basic types of assessment: I. SummativeII. Formative [1]

Measured :Qualitative and Quantitative: Putting the results into a metric context .

Qualitatively and subjectively Quantitatively and objectively :With the aid of a questionnaire

study..[2]

1-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Press. pp 9-14.2-Brender, J., Handbook of Evaluation Methods for Health Informatics ,2006: Academic Press. pp 20-29.

12

Formative and Sum-mative evaluation

Formative evaluation

Throughout the systems lifecycle

It provides information for improving the system under

development

Summative evaluation

Focused on assessing the effect or outcome of the evaluation object

At a certain point of time after implementation

1.Nykanen, P., et al., Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform, 2011. 80(12): p. 815-27.

13

Quantitative & qualitative methods:

• Task analysis• Interface design• Time motion analysis• Software log• Questionnaire

Quantitative methods

• Think aloud protocol• Unstructured interviews

Qualitative methodsTo assess

• user satisfaction• user-perceived• usefulness • usability of this

framework.

Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

14

Categories of most evaluation studies:

I. CPR evaluation studies,II. Telemedicine evaluation studies,III. DSS evaluation studies.

Evaluation can be conveniently classified into: Objectivist and Subjectivist approaches . [2]

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.2- C.P. Friedman, J. Wyatt, Evaluation methods in biomedical informatics, 2nd ed., Springer Science & Business,Media, New York, US, 2006.

15

Methodology and Meth-ods:A method is based on a well-defined theory and includes a consistent set of techniques, tools and principles to organize it.

Methodology

Method

metric metric

measure measure

Method

metric

Method

Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. p 13.

16

Methodology• Methodology is supposed to:(1) provide the answer to what to do next, when to do what, and how to do it

(2) to describe the ideas behind such choices and the suppositions (for instance , the philosophical background) behind them.

A methodology must comprise:a) The basic philosophies and theories, so that a user of the methodology can

judge the validity of its useb) Perspective c) Assumptionsd) Areas of usee) Applicable methods, tools, and techniques

Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. PP14-15.

17

Complexity of evaluation in biomedical informatics

1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006: Springer .p 6.

Evaluation methodology

Computer-based information

systems

Medicine and healthcare

delivery

Evaluation in medical

informatics

18

METHODS

19

User Requirements Specifica-tion

• Analysis of Work Procedures • Assessment of Bids • Balanced Scorecard • BIKVA• Delphi: Qualitative assessment • FieM Study• Focus Group Interview• Future Workshop• Grounded Theory• Heuristic Evaluation• Interview (no standardized)• KUBI• Logical Framework Approach• Organizational Readiness

• Pardizipp• Questionnaire (no standardized)• Requirements Assessment• Risk Assessment• Social Network Analysis• Stakeholder Analysis• SWOT• Usability• Video recording• WHO: Framework for Assessment of

Strategies

20

Technical Development PhaseUsed to provide feed-back for the technical development. Balanced Score card Clinical/Diagnostic Performance Cognitive Assessment Cognitive Walkthrough Heuristic Evaluation Risk Assessment SWOT Technical Verification Think Aloud Usability

21

Assessment Methods : Adaptation Phase

• Analysis of Work Procedures• BIKVA• Clinical/Diagnostic Performance• Cognitive Assessment• Cognitive Walkthrough• Equity Implementation Model• Field Study• Focus Group Interview• Functionality Assessment• Grounded Theory• Heuristic Evaluation• Interview (nonstandardized)• Prospective Time Series• Questionnaire (nonstandardized)

• RCT, Randomized Controlled Trial• Risk Assessment• Root Causes Analysis • Social Network Analysis• SWOT• Technical Verification• Think Aloud • Usability• User Acceptance and Satisfaction• Videorecording

Real operational assessment can take place.

22

Assessment Methods: Evaluation Phase

1. Analysis of Work Procedures2. Balanced Score card3. BIKVA4. Clinical/Diagnostic Performance5. Cognitive Assessment6. Cognitive Walkthrough7. Delphi8. Equity Implementation Model9. FieMStudy10. Focus Group Interview11. Functionality Assessment12. Grounded Theory13. Heuristic Evaluation14. Impact Assessment15. Interview (nonstandardized)

16. KUBI17. Prospective Time Series 18. Questionnaire (nonstandardized)19. RCT, Randomized Controlled Trial20. Risk Assessment21. Root Causes Analysis22. Social Network Analysis23. Stakeholder Analysis24. SWOT25. Technical Verification26. Think Aloud27. Usability28. User Acceptance and Satisfaction 29. Videorecording30. WHO: Framework for Assessment of

Strategies

23

• Usability methods, such as heuristic evaluation, cognitive walk-throughs , RCT and user testing, are increasingly used to evaluate and improve the design of clinical software applications.

• Evaluation studies do not focus solely on the structure and function of information resources; they also address their impact on persons who are customarily users of these resources and on the outcomes of users’ interactions with them to understand users’ actions.

1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006: Springer .p 6.2- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

24

1- Analysis of Work Procedures• Assessing what actually happens compared to the expectations.

They may clearly have a role as part of an evaluation study.

• Some well-known options :

1. The Learning Organization2. Enterprise modeling 3. Business Process Reengineering4. Use Cases and scenarios5. Total Quality Management6. Health Technology Assessment (HTA)7. Computer-Supported Cooperative Work (CSCW)8. Cognitive Task Analysis

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

25

Analysis of Work Proce-dures

*Note: The use of diagramming techniques and other forms of graphical modeling requires experience and understanding the principles and procedures involved.

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.1. Ammenwerth, E., et al., Systems analysis in health care: framework and example. Methods Inf Med, 2002. 41(2): p. 134-40.

Applied in a health care setting

Provide clear

structure: Framework

• Potential Views• Level of

systems

Systems analyses in a health care environment

We can use Diagramming techniques

26

Enterprise ModelingEnterprise Architecture (EA) is a strategic activity and

planning tool for an enterprise, which facilitates decision-making by enabling a conceptual view of the enterprise. The main objective of an EA approach is to define the layout of organizational components and relationships among them.

27

Business Process Modeling

28

Use Case and Scenario

29

Health Technology Assessment (HTA):

• HTA is concerned with the systematic evaluation of the consequences of the adoption and use of new health technologies and improving the evi-dence on existing technologies. [1,2]

• One of the basic lessons learned in the area of HCI is that usability evaluation should start early in the design process

• Goodman and Ahn (1999) provide a basic overview of HTA principles and methods. [3]

1. O'Reilly, D., K. Campbell, and R. Goeree, Basics of health technology assessment. Methods Mol Biol, 2009. 473: p. 263-83.

2. Stevens, A., R. Milne, and A. Burls, Health technology assessment: history and demand. J Public Health Med, 2003. 25(2): p. 98-101.

3. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Aca-demic Press.

30

2- Balanced Scorecard • Created in 1992 by Drs. Robert S. Kaplan and David P.

Norton, the Balanced Scorecard (BSC) is a revolutionary way to handle strategy management.

• Balancing focus areas by means of a set of indicators for a set of strategic objectives.

• The Balanced Scorecard is a tool used to measure an organization’s activities and initiatives against its Vision, Mission and Values as outlined in its Strate-gic Plan.

1. http://www.hrh.ca/balancedscorecard, Humber River Hospital, 10-20-20162. https://2gc.eu/- Balance Scorecards reports-10-20-2016-2GC is a strategic execution consultancy with particular experience in implementing the latest generation Balanced Scorecards. With a global client list,

31

Balanced Scorecard is a Man-agement System

Not just a measurement system

You do not just measure

your heart beat

You use the heart rate monitor

to manage your exercise

33

3-BIKVA :User Involvement in Quality Development A tool for making critical, subjective decisions about an

existing practice.

1. Identifying user satisfaction or dissatisfaction.2. Summarizing the information in group interview. 3. User encouraged to identify the reasons behind the incidents and

interactions referred to.4.Interview management in iterative process with the objective of

clarifying all the disparities that relate to the issues of quality identified by the users.

5.conclusion will then be presented to the decision makers for an assessment


34

BIKVA :User Involvement in Quality Development

Developed in Denmark and it is an evaluation and quality enhancement method .

The evaluation method is interviewing. The evaluation process starts from the clients, then moves to the front-line staff (employees in direct contact with the clients) and finally ends to managers and politicians. Clients are asked to express and justify "why they are satisfied or dissatis-fied" with the services offered.


(2016). "Linkki-hanke - yhteys työelämään." Retrieved 11/9/2016, 2016, from :http://www.palmenia.helsinki.fi/linkki/.

35

3- Questionnaires Advantage:• most people can manage it Different Types :A. Open questionsB. Checklist questionsC. The Likert scale consisting of a bar with fields to tick on a scaleD. Multipoint scaleE. Semantic differential scale, which in tabular form uses columns

with a value scale (for instance, "extremely", "very“,….) F. Categorical scale Need verification : need thorough scientific validation of the ques-

tionnaire in order to provide results leading to optimal action

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. p 164

36

Questionnaires:

• Questionnaires contain items that are pre-determined by the investigators and consequently are of limited value in identi-fying new or emergent issues in the use of a system that the investigators have not previously thought of. [1]

1.Kushniruk, A.W. and V.L. Patel, Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform, 2004. 37(1): p. 56-76.

37

4- Clinical/Diagnostic Per-formance Measurement of the diagnostic performance in measures of accuracy

and precision for IT-based expert systems and decision-support systems before the system is implemented.

Measurement is doing in Technical Development phase but it can continue during the operational phase (the Adaptation Phase and Evolution Phase )

The clinical performance of the systems (for diagnostic, prognostic , screening tasks, etc.) is typically measured with measures from medicine, such as accuracy, precision, sensitivity, specificity, and predictive values.

1.Kaplan, B., Evaluating informatics applications—clinical decision systems. Interna-tional Journal of Medical Informatics, 2001. 64: p. 15-37.2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

39

*Point*

• Kaplan reviews the literature (only for the years 1997- 1998) on clinical performance of decision-support systems for the period and includes other review articles on this subject and he concluded that most studies use an experimental or randomized controlled clinical trials design (RCT) to assess system performance.

1.Kaplan, B., Evaluating informatics applications—clinical decision systems. International Journal of Medical Informatics, 2001. 64: p. 15-37.

40 40

5- RCTRANDOMIZED CONTROLLED TRIAL

RCT is a trial in which subjects are randomly assigned to two groups: one (the experimental group) receiving the intervention that is beingtested, and the other (the comparison group or controls) receiving an

alternative treatment. The two groups are then followed up to see if any differences between the result. This helps in assessing the effectiveness of the intervention.

41 41

Hierarchy of Study Types

Descriptive•Case report•Case series•Survey

Analytic

Observational•Cross sectional•Case-control•Cohort studies

Experimental•Randomized controlled trials

Strength of evidence for causality between a risk factor and outcome

42 42

انواع مطالعاتتوصيفي تحليلي

مشاهده اي ايمداخله

كارآزمايي باليني

كارآزمايي اجتماعي

كارآزمايي ميداني

مقطعي

مورد شاهدي

كوهورت

اكولوژيك

گزارش مورد

گزارش موارد

Goal:To test

hypotheses about the

determinants of disease

In contrast, the goal of intervention studies is to test the efficacy of

specific treatments or preventive measures by assigning individual

subjects to one of two or more treatment or prevention options

43

5-RCT, Randomized Controlled Trial Randomized controlled trials are the ideal study design to

evaluate the effectiveness of health-care interventions.

RCT is used to identify marginal differences between two or more types of treatment.

Useful in assessment studies of IT-based solutions for decision-support systems and expert systems, only to a limited degree for other types of systems.

1.Navaneethan, S.D., et al., How to design a randomized controlled trial. Nephrology (Carlton), 2010. 15(8): p. 732-9.2. Bothwell, L.E. and S.H. Podolsky, The Emergence of the Randomized, Controlled Trial. N Engl J Med, 2016. 375(6): p. 501-4.P 172

3- 1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

44

All RCTs are controlled clinical trialBUT

Not all controlled trials are RCTs

Randomization

simple cluster stratified

45

5-RCT, Randomized Controlled Trial An RCT is the most rigorous scientific method for

evaluating the effectiveness of health care interventions.

However, bias could arise when there are flaws in the design and management of a trial.

1.Akobeng, A., Understanding randomized controlled trials. Archives of Disease in Childhood, 2005. 90(8): p. 840-844.

46

RCT

اثربخشی•عوارض•

انتخاب بهترين روش پيشگيرييا درمان

براي رسيدن به هدف

47

نمونه ها به روش تصادفی ساده به گروههای مورد و شاهد تقسيم میشوند

اين روش يکی از قويترين روشهای تحقيقاتی درعلوم پزشکی میباشد

سه ويژگی مهم دارد که عبارتند از : تصادفی بودن انتساب نمونه ها به گروههای مورد

بررسی (Random Allocation ) .وجود گروه کنترل دخالت وکنترل محقق ومواجهه نمونه ها با علت

فرضی.

Randomization

48

National Cancer Institute at the National Institutes of Health; https://www.cancer.gov/about-cancer/treatment/clinical-trials/what-are-trials/randomization; 11/2/2016.

https://www.cancer.gov/about-cancer/treatment/clinical-trials/what-are-trials/randomization




49

RCT usage:I. By some patients being treated under the old

system and others under the new system, or equally with regard to staff;

II. By some departments keeping the old system and other department getting the new system;

III. Comparing similar departments in different hospitals.


50

مناسب مواردی چه برایRCTنیست؟

• Testing the etiology of disease• rare outcome • Low participation• Exclusion• Low cost

51

RCT IS NOT suitable for: * ETIOLOGY AND CLINICAL COURSE smoking and cancer

* RARE & PROLONGED OUTCOME

52

CASP

RCTs - a checklist• Good randomisation procedures• Patients blind to treatment• Clinicians blind to treatment• All participants followed up• All participants analysed in the groups to

which they were randomised (intention to treat)

53

6- Delphi• Delphi may be characterized as a method for structuring

a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem. [1,3]

• When judgment is necessary with controlled opinion feedback: we use Delphi

• Qualitative assessment of an effect • Method for the prediction of the future • Key advantage : avoids direct confrontation of the

experts. • Tool for expert problem solving [1,2]1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.2.Okoli, C. and S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations and applications. Information & Management, 2004. 42(1): p. 15-29.

3 - : علوم در آموزش ايراني مجله اباذري، پروانه نصيرياني، خديجnه احمدي، اله فضل تحقيق، در ابزاري دلفي تكنيك - تابستان و بهار ؛ 8)1 :(1387185پزشكي،

54

Delphi:• Delphi survey, including guidelines for data collection,

data analysis and reporting of results .

• Three ways of Delphi methods: 1- Classic Delphi, 2- Policy Delphi, 3- Decision Delphi

• Preparation of a questionnaire, including the perils and pitfalls associated with a Questionnaire approach

1.Okoli, C. and S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations and applications. Information & Management, 2004. 42(1): p. 15-29.2.Kushniruk, A.W. and V.L. Patel, Cognitive and usability engineering methods for the evaluation of clinical informa-tion systems. J Biomed Inform, 2004. 37(1): p. 56-76.

3 : علوم. در آموزش ايراني مجله اباذري، پروانه نصيرياني، خديجه احمدي، اله فضل تحقيق، دnر ابزاري دلفي تكنيك - تابستان و بهار ؛ 8)1 :(1387185پزشكي،

56

Experts selection:

57

مراحل اجرایی روش دلفی :

ول روش دلفیnنظرات شرکت کنندگان استفاده از خرد جمعی و کثرت گرایی اصمیباشد.

تعداد شرکت در روش دلفی با توجه به دامنه و گستره موضوع و منابع قابل دسترسخواهد بود.کنندگان متفاوت

وش تالش بر این است که شرکت کنندگان برای همnمانند.ناشناس در این رnباقی ب

کرار میباشد که در طی این مراحل سعی بر آن است کهnاین روش شامل چندین دوره ترا به عمل آورده، مخالفت و ناسازگاری به حداکثر استفاده از نظرات گروهی

پایان می یابد.راند 3یا 2معموال در حداقل برسد.

به همراه فیلد باز فیلد باز و یا نیمه س}اختار مند چک لیست مرحله اول به صورت(جهت پیشنهاد آیتم های جدید( طراحی می گردد.

طراحی می گردد. )بدون فیلد باز(ساختار مند چک لیست سایر مراحل به صورت

استفاده می گردد.تایی لیکرت 5معیار به طور معمول از

نتایج دور تحلیل خواهد شد و نتایج دور اول با ترکیبی از روش کمی و کیفیخواهندn داشت و از فنون رتبه بندی و درجه دوم و دورهای بعد ماهیتی کامال کمی

بندی تحلیل میشونnد.

.به طور معمول جهت بررسی میزان توافق از میانه و یا میانگین استفاده می گردد

تعیین می گردد.با نظر متخصصین آستانه میزان توافق

58

6- Delphi

1.Friedman C., W.J., Evaluation methods in biomedical informatics. 2006: Springer .p 182.

59

7-Usability

• Usability evaluation is a method for identifying specific problems with IT products and specifically focuses on the interaction between the user and task in a defined environment.[1,4]

• The usability evaluation methods are so various which makes the choice of the appropriate one(s) a difficult task for evaluators. Usability factors are various features that are used to measure how easy systems are in support-ing users’ tasks.[2,3]

1.Brown Iii, W., et al., Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. Journal of Biomedical Informatics, 2013. 46(6): p. 1080-1087.2.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive systems. 2016: p. 246-251.3.Borycki, E., et al., Usability Methods for Ensuring Health Information Technology Safety: Evidence-Based Approaches Contribution of the IMIA Working Group Health Informaticsfor Patient Safety. IMIA Yearbook of Medical Informatics, 2013: p. 20-27.4. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

60

USABILITY EVALUATION METHODS FOR INTERACTIVE ADAPTIVE SYSTEMS:

• Cognitive Walkthrough• Heuristic Evaluation• Focus Group• User-as-wizard• Task experiment based• Simulated users

1.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive systems. 2016: p. 246-251.

61

Usability EvaluationMethods

Advantages Disadvantages

Cognitivewalkthrough

(+) Makes predictionsabout the design withoutinvolving users,(+) Can be used early in the preliminary eval-uation phase.

(–)Certain adaptation aspects not covered,(–)Time consuming.

Heuristic evaluation (+) Quick and low-costevaluation method,(+) Can be done early in the IAS developmentprocess

(–) Evaluators mustbe experts,(–) Need to chooseThe appropriateheuristics.

Focus group (+) Fast to conduct,(+) Can provide largeamounts of data in short time

(–) Support onlysubjective opinions(–) Depends onexperiencedmoderator.

1.Dhouib, A., et al., A classification and comparison of usability evaluation methods for interactive adaptive systems. 2016: p. 246-251.

62

8- Cognitive Assessment • Task-oriented

• Assessment of the cognitive aspects of the interaction between an IT system and its users

• Cognitive aspects deal with the extent to which the system functions in accordance with the way the users

• Think and work in respect of user interaction with the IT system in general and with the user dialogue in par-ticular .


63

9- Cognitive Walkthrough• Cognitive Walkthroughs an analytical method designed to evaluate

usability • Assessment of user 'friendliness' on the basis of a system design,

from specification, to muck-ups and early prototypes of the system, aimed at judging how well the system complies with the users' way of thinking

• The cognitive walkthrough is a formalized way of imagining peo-ple’s thoughts and actions when they use an interface for the first time.

1. Wharton, C., et al., The cognitive walkthrough method: a practitioner's guide, in Usability inspection methods, N. Jakob and L.M. Robert, Editors. 1994, John Wiley & Sons, Inc. p. 105-140.

2. A. W. Kushniruk, A. W. Kushniruk, E. M. Borycki-Human, Social, and Organizational As-pects of Health Information Systems-IGI Global, Premier Reference Source (2008)

3. Xiao, T., Broxham, W., Stitzlein, C., Croll, J., & Sanderson, P. Two human-centered approaches to health informatics: Cognitive systems engineering and usability. in Proceedings of the WCC IFIP-IMIA International eHealth Joint Conference. 2010. Brisbane, Qld.

64

Cognitive Walkthrough• CW is an inspection method for evaluating usability in a user in-

terface.• The questions are aids to simulating the user’s cognitive process:

(1)Will the user be trying to achieve the right effect?(2)Will the user discover that the correct action is available?(3)Will the user associate the correct action with the desired effect?(4)If the correct action is performed, will the user see that progress is being made? answered with YES or NO and reasons why the user will succeed or fail in performing the action (“failure/success stories”). [1,2]

1.Blig, L.-O. and A.-L. Osvalder, Enhanced cognitive walkthrough: development of the cognitive walkthrough method to better predict, identify, and present usability problems. Adv. in Hum.-Comp. Int., 2013. 2013: p. 9-17.2.Wharton, C., et al., The cognitive walkthrough method: a practitioner's guide, in Usability inspec-tion methods, N. Jakob and L.M. Robert, Editors. 1994, John Wiley & Sons, Inc. p. 105-140.

65

CW• The method focuses on simplicity in learning, especially

through exploratory learning. CW has been employed to evaluate medical equipment, such as clinical informationsystems, patient information systems, clinical order systems , dialysis machines, and patient surveillance systems .[1]

1.Blig, L.-O. and A.-L. Osvalder , Enhanced cognitive walkthrough: development of the cognitive walkthrough method to better predict, identify, and present usability problems.

Adv. in Hum.-Comp. Int., 2013. 2013: p. 9-17.

66

10-Focus Group Interview • A focus group is a form of qualitative research in which

a group of people are asked about their perceptions, opinions, beliefs, and attitudes towards a product.

• The moderator speaks very little, and encourages the group to generate the information required by stimulating discussion through terse provocative statements.

Challenge: analyzing qualitative data

1.Rabiee, F., Focus-group interview and data analysis. Proc Nutr Soc, 2004. 63(4): p. 655-60.2.Chapter 5: Personal Interviews. Marketing research and information systems. (Marketing and Agribusiness Texts 2016 [cited 2016 10/31/2016]; Available from: http://www.fao.org/docrep/w3241e/w3241e06.htm.

67

Time of focus group inter-view:

During the early analysis phases

During operations

68

The focus used for eliciting descriptive data from population sub-groups

Usually, a group of eight to twelve persons are gathered together for a group interview or discussion on a focused topic.

Focus groups are widely used in the investigation of applied-research problems and are recognized as a distinct research method.

The method enables researchers to generate new hypotheses; To explore intermediate variables as a means of explaining certain

relationships found in survey data.

1. Bender, D.E. and D. Ewbank, The focus group as a tool for health research: issues in design and analysis. Health Transit Rev, 1994. 4(1): p. 63-80.

69

11-Equity Implementation Model (EIM)

• Examine users' reaction to the implementation of a new system

• It may be applied as a measuring instrument to provide the deci-sion-making basis relation to policy planning and priority setting, as well as for examination of barriers and drivers at the implementa-tion of change.

• The EIM helps provide a theory-based understanding for collecting and reviewing users' reactions to, and acceptance or rejection of, a new technology or system.

1. Lauer, T.W., K. Joshi, and T. Browdy, Use of the equity implementation model to review clinical system implementation efforts: a case report. J Am Med Inform Assoc, 2000. 7(1): p. 91-102.2. Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

70

• The method provides a means for explanation of systemimplementation events, leading to an understanding of user resistance or acceptance of the new technology.

• It may be applied as a measuring instrument to provide the deci-sion-making basis relation to policy planning and priority setting, as well as for examination of barriers and drivers at the implemen-tation of change.

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. P 110-

71

12- Functionality Assessment

1. Validation of objectives fulfillment 2. Impact Assessment 3. Identification of problems

• It is best suited to qualitative studies


72

13-FieMStudy Observation of an organization to identify its practices

and to expose mechanisms that control change.

This method is widely used in psychology, sociology, anthropology, and so on that is, in professions that deal with different perspectives of human factors - to identify what goes on and how


73

14-Grounded Theory• Grounded Theory is a supportive analytical method for

data acquisition methods that generate textual data, some open Questionnaire methods, and Interviews.

• This method is subjective.

• In a social context through an iterative and comparative process in 4 phase:

Open coding Axial coding Selective coding Theoretical coding


74

15-Heuristic Evaluation• Used when no other realizable possibilities exist • This method can be used for nearly everything but in practice it is

most commonly used for the assessment of user interfaces• Heuristic evaluation, an established evaluation method in the field

of Human-Computer Interaction, was originally proposed by Jakob Nielsen .

Five phase:1. Selection of appropriate heuristics, 2. An individual and independent inspection of the design using heuristics to

identify features that conflict with some aspect of best practice,3. Editing the joint material to identify and resolve duplicates and related find-

ings,4. Prioritization to identify severities5. Analysis and report writing 1.Westbrook, J. and J.I. Westbrook, Information Technology in Health Care 2007: Pro-ceedings of the 3rd International Conference on Information Technology in Health Care: Socio-Technical Approaches. 2007: IOS Press . pp 205-2072- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.

76

Zhang and colleagues incorporated Nielsen’s 10 heuristics , Shneiderman’s eight golden rules , and the results of their research to formulate a list of 14 heuristics.

1.Allen, M., et al., Heuristic evaluation of paper-based Web pages: A simplified inspec-tion usability methodology. Journal of Biomedical Informatics, 2006. 39(4): p. 412-423.

77

16- Impact Assessment

• Measurement of the effect plus assess-ment of side effect


78

17- Interview (Nonstandardized)

• For qualitative studies of subjective as well as objective circumstances.


79

18- KUBI• Translated: Quality development through user involvement • It has many points in common with the Balanced Scorecard

method


80

19- SWOT• Allows the assessment of an organization from a

neutral perspective through a detailed discussion of the organization’s strengths, weaknesses, op-portunities and threats.

81

SWOT

• The method is useful as a brainstorming method to elucidate and evaluate each of the aspects brought forward.


82

20- Think Aloud • Think Aloud is a method that requires

users to speak as they interact with an IT-based system to solve a problem or per-form a task.


83

• Thinking aloud technique dates back to the works of experimental psychology and was first ever described by Karl Duncker (1945) while he studied productive thinking.

• TAP is popularly used by usability researchers today. But what do researchers think they get from TAP? Is it right to assume that there is a one-to-one mapping between verbal protocols and ‘pure data’?

• TAP adds cognitive load on users and can hinder primary tasks (Preece, 1994). So how do the users experience it? What do users think of it?

Research Paper: Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes on in people's heads?: Reflections on the think-aloud technique. In Proceedings of the second Nordic conference on Human-computer interaction (NordiCHI '02). ACM, New York, NY.

84

• Thinking is much more that what can be explicitly expressed in words.

•To get access to human cognitive processes, a way forward may be to develop a practice of introspection; to expand our knowledge about the reflective activity of the user in the expert-guided think aloud session.

•The authors argue that access to subjective experience is possible in terms of introspection where user has to become a participant in the analysis of his or her own cognitive processes.

• The paper suggests that use of think aloud should have, as a prerequisite, explicit descriptions of design, test procedure and framework for analysis.

85

21- Prospective Time Se-ries Measurement of a development trend Time series Before-and-after studies Measurement of a number of measures over time shows how an ac-

tivity or an outcome changes as a function of either time alone or as a function of different initiatives

This method requires control all over time

Carried out as a matched-pair design in: Controlled studies including RCTs Simple before-and-after study In cohort studies with the follow-up of cases over time In decision support system with before-after study

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press. P 160.

86

22-Risk Assessment• Monitoring of risk factors in a

development or assessment project to make it possible to take preemptive action.

• Risk assessment, risk management, and risk control can be handled either retrospectively or prospectively.


87

23-Root Causes Analysis

• Exploration of what, how, and why a given incident occurred to identify the root causes of undesirable events.


88

24-Social Network Analysis• Social network analysis is carried out by means of dia-

gramming techniques in relationships between elements within an organization (such as individuals, professions, departments or other organizations)

• Most commonly applied to help improve the effective-ness and efficiency of decision making processes in commercial organizations.

• The potential value of SNA to measure team function and use the information to improve working processes was another finding of studies in different settings

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.2- Chambers, D., Wilson, P., Thompson, C., & Harden, M. (2012). Social Network Analysis in Healthcare Settings: A Systematic Scoping Review. PLoS ONE, 7(8), e41911. http://doi.org/10.1371/journal.pone.0041911

89

Example of social network: the case of the eye care programme in the Brong Ahafo region, January 2010.

Karl Blanchet, and Philip James Health Policy Plan. 2011;heapol.czr055

Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine © The Author 2011; all rights reserved.

90

25-Stakeholder Analysis • Assessment of stakeholder features and their inner dynamics, aim-

ing to identify participants for the completion of a given task, a problem-solving activity or a project.

Namazzi, G., et al., Stakeholder analysis for a maternal and newborn health project in Eastern Uganda. BMC Pregnancy Childbirth, 2013. 13: p. 58.

91

Stakeholder Analysis. Service quality 2016 [cited 2016 11/2/2016]; Available from: http://

asq.org/service/body-of-knowledge/tools-stakeholder-analysis.

92

26-Technical Verification • Functions are present, work correctly and are in

compliance with the agreement.

A. Completeness of the functionality B. Correctness of the functionality C. Coherence of the functionality D. Consistency of the functionality E. Interconnectivity of the functionality F. Reliability G. Performance


93

27-User Acceptance and Satisfaction Assessment of user opinions, attitudes, and perceptions of an IT

system during daily operation Most common methods : Interview techniques : Interviews or Focus-Group Interviews. Questionnaires: most commonly used when measuring user satis-

faction. The Equity Implementation Model

*User-satisfaction questionnaires Aim: verify and correct for aspects such as reliability, predictive value, accuracy and content validity and internal validity.


94

28-Videorecording Video recording is a capability for recording videos of every new Automate test session that is requested on BrowserStack. While your test is running, you can see a live screencast that is recorded and saved with a proper name that can be accessed and downloaded later.

https://www.browserstack.com/question/651

95

29-WHO: Framework for Assessment of Strate-gies The method may be applied for assessment of different

(development) strategies, either individually or as a comparative analysis.

The method is particularly relevant early in the explorative Phase for exploring the feasibility of basic ideas and solutions.

1- Brender, J., Handbook of Evaluation Methods for Health Informatics 2006: Academic Press.p 222

96

30-WHO: Framework for Assessment of Strategies

97

Conclusion

Literature reviews

98

Literature reviews:

• Evaluation studies have been performed with different aims and ob-jectives, and are concerned with different domains and areas of fo-cus.

1.Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in Healthcare Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-432.

99

Literature reviews about ICT Evalu-ation Most important study conducted by Kapalan about CDSS Evaluation[1].• Conclusion of 27 reviews : RCT is standard method of evaluation approach for

CDSS.• Problems of RCT: Do not answer questions such as why some systems tend to be

used while others are not•Unrecognized ineffectiveness reason of CDSS.

1,035 articles reviewed by Ammenwerth and Keizer’s from 1982 to 2002• Concluded that the number of evaluation studies in the area of medical informatics is

rising significantly.• appropriateness of patient care, efficiency of patient care, user satisfaction, and software

quality.• The quality of care processes and patient outcomes was found to have increased.• Interpreted this shift as a sign of the maturation of evaluation research in medical

informatics.

1. Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in Health-care Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-432.

100

Delpierre et al reviewed 26 articles about CBPRS , from January 2000 to March 2003.

• Increased satisfaction of users and patients• Could lead to significant changes in medical practice.• Most of the studies did not include qualitative factors such as

characteristics of the disease and the tool

Rahimi, B. and V. Vimarlund, Methods to Evaluate Health information Systems in Health-care Settings: A Literature Review. Journal of Medical Systems, 2007. 31(5): p. 397-432.

101

• Some of the studies in this review evaluate the effects of the new system’s implementation on the quality of work performance, such as user job performance , computer knowledge, and investigation of skill among other users

• Most of the studies included in this paper have used survey methodology as their research method

• Some of the studies used a clinical trial or cohort study to research the systems 'outputs.

• No studies have been conducted to explore the impact of IT on the system as a whole.

103

The End

104

Free PPT Chart : ALLPPT.com

ALLPPT.com _ Free Powerpoint Templates, Diagrams and Charts

Your own sub headline

Text hereAdd text Add

Text here Text hereAdd text add text add text add text add text add text add text add text Text here Text hereAdd text add text add text add text

Text here Text hereAdd text add text add text

add text add text add text add text

Text here Text hereAdd text add text add text add text add text add text add text add text

Text here Text hereAdd text add text add text add text

http://www.free-powerpoint-templates-design.com/free-powerpoint-diagrams-design/