modifying the mars scale to evaluate health applications€¦ · •developed at queensland...

1

Modifying the MARS Scale to Evaluate Health Applications

Session #107, February 13, 2019

Speaker: Ryan Romero, MPH Student, University of Florida

Rick Kates, Clinical Assistant Professor, University of Florida

2

Ryan Romero, MPH Student

Has no real or apparent conflicts of interest to report.

Frederick Kates, PhDHas no real or apparent conflicts of interest to report.

Conflict of Interest

3

1. Introduction to MARS

– A. Sections of the MARS Scale

– B. Application Specific Measures

– C. Validation of Content Specific Measures

2. Study Specific Populations and Our Application of MARS

– A. The deaf and Hard of Hearing Community

– B. Use of Hard of Hearing Applications

– C. App Specific Measures Developed for Target Population

– D. Study Characteristics

3. New Criteria For Application Specific Measures Using MARS

– A. Identification of a content expert in subject field (with example)

– B. Use of content expert or population member to help develop content specific measures

– C. Use of content expert to gauge specific interests and needs of population

– D. Short wrap up discussion

Agenda

4

• Design content-specific criteria to evaluate mobile

health applications

• Identify and engage an appropriate content

expert

• Identify population-specific needs and implement

criteria into rating scale

Learning Objectives

5

• Content Expert: Knowledge expert in a particular field of study that has some professional involvement in the subject [1]

• Application Specific Measures (Content Specific): Evaluation metrics designed to gauge the value of specific features towards an intended population/ function [2]

• Appropriateness: Relevance of features to intended users

• MARS (Mobile Application Rating Scale): Quantitative (1-5) scale used to evaluate mHealth applications [2]

Key Terms

6

Join at kahoot.it

1. Health Infinity 2. Carb Manager 3. MyNetDiary

Which mHealth App is the Best?

7

1. Health Infinity

8

2. Carb Manager

9

3. MyNetDiary

10

• The Mobile Application Rating Scale (MARS) [2]

• Developed at Queensland University of Technology

– Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko, Dian Tjondronegoro, MadhavanMani

• Provides a standardized approach to comparing and rating health applications for mobile devices (iPhone, iPad, Android, Android Tablet)

• Allows for modularity as key part of design

• Frequently used to evaluate mHealth applications

Introduction to MARS

11

Sections of the MARS Scale[2]

Engagement

5 subsections

Engagement

5 subsections

Functionality4 subsections

Functionality4 subsections

Aesthetics

3 subsections

Aesthetics

3 subsections

Information

7 subsections

Information

7 subsections

Subjective

Quality*

Subjective

Quality*

12

Alpha test of interrater agreement is preformed to ensure acceptable levels of agreement [3]. Sections where there is a lack of agreement or uncertainty are discussed and resolved by consultation of the content expert and reevaluation.

Training and Validation of Raters

13

• MARS allows for custom measures intrinsically

– Tells us about app quality

– Tells us about appropriateness and intended fit

• Use of section as indicator of status vs indicator for change*

– Section can be used to both address quality and declare the need or desire for specific feature

• Content development and developer feedback

– Use of content expert to guide study’s intent

– Feedback to developers

– Empirical data to support stance

Content Specific Measures

14

Study Specific Populations and Our Application of the

MARS

15

New Translation Technologies

16

• The Deaf and Hard-of-Hearing community- 7 million members of HOH community[6]

• Electronics as part of social interface capability

• Electronics used to communicate with family and friends

• Necessity for evaluation of hard-of-hearing applications

Hard of Hearing Apps & Our Study

17

• Preliminary Literature Review

– 37 million with hearing difficulties[7]

– Needs of population over 60 years old[8]

• Decision to Include Content Expert

• Inclusion and Exclusion Criteria

– Need to determine appropriateness[9]

– Mitigate risk

• Results

Study Characteristics

18

• Application features

• Is technology likely to be used?

• Inclusion and Exclusion Criteria

• Multi category approach to population specific applications

– Text to Speech

– ASL Translator

– Hard-of-Hearing Assistant

Hard of Hearing Applications

19

Content Expert

Stephen J. Hardy, IIM.Ed.

Lecturer, University of Florida

College of Public Health and Health

Professions

Fluent in Deaf Culture, ASL, and Deaf

Studies

20

App-Specific Measure #1

21

App-Specific Measure #2

22

App-Specific Measure #3

[4]

23

App-Specific Measure #4

24

Related Study

[5]

25

• 5 / 21 applications no longer available for evaluation after follow up

– Simply disappeared

– Indicator of status of application field

– Potential lack of content validity

– Niche product does not necessarily entail less competition

Application Turnover

26

• Please take out your smartphone or device and enter the activity at: Kahoot.it

• Winner will receive small non-monetary prize to keep it interesting

Time to Get Involved!

27

• Three important topics:

1. Use of content experts as guide for project

2. Use of content experts to achieve validity of experiment design

3. Approaching a content expert for consultation

Content Experts and Design Validity

28

• What qualifications should my content expert have?

– How should they relate to the topic we are studying?

• How can they be used to the advantage of the study?

– Guide for purpose

– Who does study effect?

– Discussion section

• Use of multiple content experts

Identification of a Content Expert

29

• Knowledge on use of content experts in experimental design and analysis

• Recognition of importance of content validity and designing a study to help the subject it focuses on

• Recommendation of quality apps based on MARS

How Can This Information Help YOU?

30

• Discussion on how this information relates to your focus

• Using this information to develop a study

• How health information professionals can use this

information to design better applications

Discussion and Q&A

31

• Ryan Romero, MPH Student

– ryan.romero1000@ufl.edu

• Rick Kates, PhD

– kates.rick@phhp.ufl.edu

Please feel free to come up and speak with us after the presentation!

Speaker Contact

32

1. Grant, J.S., & Davis, L.T. (1997). Selection and use of

content experts in instrument development. Research

in Nursing & Health, 20, 269–274.

2. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M

Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps JMIR

Mhealth Uhealth 2015;3(1):e27. URL: https://mhealth.jmir.org/2015/1/e27

3. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull

19790101;86(2):420. [doi: 10.1037/0033-2909.86.2.420]

4. SkoogMusic. The Skoogmusic Skwitch. Digital Image. Accessed from

http://skoogmusic.com/skwitch/

5. Arellano, Patricia, Bochinski, Janet, Elias, Beth, Houser, Shannon, Martin, Thomas, Head, Hank.

2012. Selecting a Mobile App: Evaluating the Usability of Medical Applications. mHIMSS App

Usability Work Group.

6. Carroll DD, Courtney-Long EA, Stevens AC, Sloan ML, Lullo C, Visser SN, Fox MH, Armour BS,

Campbell VA, Brown DR, Dorn JM. Vital signs: disability and physical activity--United States,

2009-2012. MMWR Morb Mortal Wkly Rep 2014 May 9;63(18):407–413. PMID:24807240

7. Ross, Mitchell, Karchmer, Michael. (2004). Chasing the Mythical Ten Percent: Parental Hearing Status of Deaf and Hard of Hearing Students in the

United States. Sign Language Studies, (4)2; 138-163.

8. Hoffman, H. J., Dobie, R. A., Losonczy, K. G., Themann, C. L., & Flamme, G. A. (2017). Declining Prevalence of Hearing Loss in US Adults Aged 20

to 69 Years. JAMA otolaryngology-- head & neck surgery, 143(3), 274-285.

9. Lewis, T. L., & Wyatt, J. C. (2014). mHealth and mobile medical Apps: a framework to assess risk and promote safer use. Journal of medical Internet research, 16(9), e210.

doi:10.2196/jmir.3133

References