preliminary results from a survey on the use of metrics and evaluation strategies among mhealth...

Preliminary results from a survey on Preliminary results from a survey on the use of metrics and evaluation the use of metrics and evaluation

strategies among mHealth projectsstrategies among mHealth projects

Garrett Mehl, Franz Allmayer, Heli Bathija Department of Reproductive Health and Research, WHO, Switzerland

Patricia Mechael, Nadi KaongaCenter for Global Health and Economic Development at the Earth Institute, Columbia University

Survey Instrument• 50 multiple choice and open ended questions

• ~ 15 minutes for each project

• Survey topics

• Health focus areas, strategic approaches

• Specific objectives, types of monitoring, approach to evaluation

• Project phase, duration, level of support

• Self-identified areas of need

Survey Methodology

• Jointly developed with Columbia U.

• The survey is still being carried out.

• Available as a web-survey or soft-copy for printing

• Survey was announced to various electronic forums

• Additionally, individuals were sent targeted emails.

• Twitter

Survey ReachSurvey Reach

USA:5

Peru: 7

Haiti:2Mexico: 3

India:6

Pakistan:4Kenya:11Uganda:10South Africa:4Tanzania:5Ghana:3Nigeria:5Malawi: 2

Philipines:4

All countries highlighted contain an mHealth project covered by the survey. Countries with multiple projects are specified.

Jordan

Project duration in sample

Projects by health focus

Current Phase of projects

Current Phase

Needs assessment 8.8%

Usability testing 8.8%

Pilot not for scaling 7.4%

Pilot for scaling 57.4%

Large scale implementation

17.6%

Frequency of project objectives

• Client information

• Health provider information

• Increasing access to services

• Improve service quality and/or safety

• Increase client service demand

• Increase time savings

• Improve intra-provider communication

• Increase provider skills

• Reduce unneeded referrals

• Increase patient treatment compliance

• Reduce health service costs

• Improve ability to respond to crises

• Reduce stock-outs

Client focusProvider focusEfficiencies

mHealth Strategies

Clustering of mHealth objectives

StockCrisis

response

Time savingsReduce referral

Communication

Provider Skills

Lower costsQuality/

SafetyCompliance

Service DemandService access

Client info

Provider info

Domains of Measurement

Measurement domains of focus

Level of monitoring

Focus on Monitoring and Evaluation

Focus of Evaluation Assessment

Did the intervention result in improvements in:

Number of evaluation questions tracked

CostsSustainabilityBehavior changesHealth outcomesKnowledge, attitudesPerformanceQuality of CareService Utilization

Type of evaluation approach

Descriptive 29.8%

Cross-sectional 44.7%

Longitudinal 40%

Case-control 12.8%

Wedge 8.5%

Rigor of Evaluation Design

Drivers of Monitoring

Level of Monitoring

Significance

Performance Accounting

correlation coefficient

= .392(spearman’s

rho)

sig. (2-tailed)

.002

Number of funding sources

correlation coefficient

= .456(spearman’s

rho)

sig. (2-tailed)

.000

Project phase and monitoring level

Evaluation rigor by project start date

Evaluation rigor by project phase

Drivers of evaluation rigorRigor of

evaluationSignificance

Performance Accounting

correlation coefficient

= .501(spearman’s

rho)

sig. (2-tailed)

.000

Number of funding sources

correlation coefficient

= .392(spearman’s

rho)

sig. (2-tailed)

.001

Assistance requested

Assistance need by project phase

“We need a systematic approach to analyzing the

data we have collected over the past 3 years. “

Assistance Requested

“We also need to learn what is the norm for "success" in this field and

how we stack up to normal interventions vs. other mHealth

projects working on [similar] technology.”

“We need guidance on evaluation methods for

mHealth”

“We are interested in collaborative approaches

and standard indicators that will be measured across the

different mHealth programs.”

“How to assess the impact of [our] mHealth tool.”

Thank you.

For more information, or to submit your mHealth project to the survey, please send an

email to:

Dr. Garrett Mehlmehlg@who.int