"using data science to design effective precision preventative behavioral medicine" - ryan...

Using Data Science to Design Effective Precision Preventative Behavioral Medicine

Ryan QuanData ScientistOmada Health

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

3

Eric Williams

Include in PotW: omadastaff comparision to BoB







4

Eric Williams


OMADA HEALTH 5OMADA HEALTH

THE PROBLEM

6

THE SOLUTION

9

Lifestyle Change

10

58% Reduction

11

Large Body of Clinical Evidence for DPP

THE PROBLEMTHE SOLUTION

WHAT’S THE DEAL?

12

13

Behavior Change is Hard

EDUCATION

TRACKERS

CALORIE COUNTING

HEALTH COACHING

SOCIAL SUPPORT

One Dimension is Not Enough

19

INTRODUCINGThe Omada Program

OMADA HEALTH

PARTICIPANT PEERGROUP

Participants are matched into peer groups for

encouragement and healthy competition with the entire group progressing along a

shared timeline.

EASY-TO-USETECHNOLOGY

A wireless scale, pedometer, mobile apps, and interactive

portal are used to track weight loss, activity, meals,

and program accomplishments.

ONLINE INTERACTIVE LESSONS

Participants learn how to eat healthier, increase activity

levels, and overcome challenges through fun lessons

and games accessed via an online portal or mobile app.

FULL-TIME PROFESSIONALPREVENT COACH

A dedicated coach provides participants with support,

motivation, and personalized recommendations along their

Prevent journey.

22OMADA HEALTH

OVER 19 POINTS OF ENGAGEMENT PER WEEK

Group DiscussionPrevent Coach ConversationSkill ChallengeSuccess Updates

Weigh-InsExercise LogMeal TrackingLesson Completions

OMADA HEALTH 24

OMADA

26Data Science

Data Science @ Omada

27

• Use analytics, machine learning and experimentation to parameterize what works (and what doesn’t) for behavior change

• Deploy the right intervention, at the right time, for the right patient

Precision Preventative Medicine

The Data (one participant out of 100k+)

28

5

0

Weight Loss (%

)

15

10

5

0

Weight Loss (%

)

15

10

29

Private Messages• Private messages between

HC and participants• Over 1.2M messages• Data:

- Emotionally charged raw text (NLP)


5

0

Weight Loss (%

)

15

10

30

Lessons / Curriculum• Based on original DPP

curriculum• Basic nutrition/activity

information• Interactive w/ games, etc.• Data:

- Completion rates- Reading times- Comprehension


5

0

Weight Loss (%

)

15

10

31

Physical Activity Tracking• Personalized adaptive daily step goals• MyFitnessPal and auto-step tracking

integrations• Over 3.6M activities recorded• Data:

- Steps entered- Minutes of activity- Location (physical proximity triggers)


5

0

Weight Loss (%

)

15

10

32

Group Messaging / Social Support• Group communication • participant <> participant• participant <> health coach• Data:

- Emotionally charged text communication (NLP)


5

0

Weight Loss (%

)

15

10

33

Meal Tracking

• Daily food tracking• 6.2M meals tracked• Active feedback from coach• Data:

- Raw text (food)- Healthiness- Portion size- Time of day


5

0

Weight Loss (%

)

15

10

34

Weighing-in / Progress • Low-power 3G scale• Automatically uploads to

profile on progress page• Weighing in as habit

formation• 6M weigh-ins• Data:

- Weight value- Time of day


The Data



Building a Data Science Culture from the Ground Up

• Winning hearts & minds with data




36

Eric Williams


Building a data science cultureScenario:

You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science

team or experience with data driven product development.

What do you do? Two choices:




What do you do? Two choices:Choice 1: Data Police




What do you do? Two choices:Choice 2: Build a data

science culture

Building a data science culture

Lots of ways to kick this off:• Brown bag lunch presentations / journal club• Deploy visualization tools• Data education/proselytization• Pro-active analytics as a service• Start a data blog…

Building a data science culture“Plot of the Week (PotW)”

• Internal weekly email data blog • ‘Buzz-feedy’ stories using our data in interesting ways• Exposed company to opportunities with data• Mobilized support and excitement for data science

Building a data science culture“Plot of the Week (PotW)” examples:

1. All in the Family Provides view into raw data

2. Minds over Matter Illustrates potential uses of our data

3. Just Breathe Educates subject matter with data

Building a data science culture“Plot of the Week (PotW)” examples:

1. All in the Family Provides view into raw data

2. Minds over Matter Illustrates potential uses of our data

3. Just Breathe Educates subject matter with data

Our scales become ‘part of the family’ sending weight data of anyone (anything) that steps

on it

PotW: All in the Family

44

Family of two, losing weight together


45

Another family of two, losing weight together, a few trips to Sandals in there too?


46

Family of three, growing child


47

Family of three, neighborhood party


48

???







49

Eric Williams


The Data Science & Product Relationship

Product

50

Sub-optimal - data science as ‘advisor’

Data-Science

• Exploration/data mining

• Modeling• Proto-typing• Lean-experimentation

• Prioritizes engineering work

• Creates data-models• Implements experiments• Requests analysis

Recommendations/insights

Requests


Product

51

Sub-optimal - data science as ‘advisor’

Data-Science

Recommendations/insights

Requests


Data Science

52

ProductOptimal - data science as ‘driver’

• Has own backlog/engineers• Drives data-model design• Experimentation built into

Product• Data scientists can launch

production experiments

Hypothesis

Generation

Experimentation

IterationOptimization





The Paths to Personalized (Precision) Prevention

• How to target the right patient, at the right time, in the right way• Impactful experimentation: beyond the sub-group analysis

53

Eric Williams








54

Eric Williams


Impactful Experimentation

55

Pros Cons• Causality !=

Correlation• Common language of

scientific discovery

• Low N• Labor intensive, expensive• Slow• Bias in patient populations• Results often not generalizable

For over a century, the Randomized Control Trial (RCT) has been the gold standard for scientific discovery

Impactful Experimentation

56

Largest meta-analysisof DPP ~5.5k participants

>60k Omada participants

vs

At Omada, we have built Randomized Control Experiments into the product to take advantage of the measurement power of experimentation, while the digital nature of our program mitigates the cons (e.g. high N, quick iteration, inexpensive, generalized population, deeper data).

Real-time, Randomized Controlled Experiments (RCEs)

35 in-product experiments & growing quicklyThis enables:

• Rapid exploration of product hypotheses• Structured innovation and evolution of

Prevent• Massive personalization of sub-specifics of

the intervention • Centralized intervention delivery allows

coordinated optimization across all demographics and sectors

• Asking fundamental about what works, and what doesn’t, in behavior change

Robert Ellis

A "yes, and ..." on Eric's comment: playing up the duality of intimately personalized and widely scaled; "precision preventive medicine for millions" (the equivalent of Uber's "Everyone's private driver").

Eric Williams

* Centralized intervention delivery allows optimization across all demographics and sectors* Our data > 10X that of largest meta-analysis (Dunkley) and much deeper for each participant (weight, engagement, demo)* Goal is _precision preventative medicine_, the right intervention for the right individual at the right time

Sean Duffy

+1. Thanks for guidance on TPs. Sami, maybe see if there's a way to spruce up slide, but no need for content changes.

Sami Lukens

spruced.

Personalization through ExperimentationSubgroup Analysis

Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”

Broad cuts lose resolution to subtle changes

Guided by heuristics (e.g. demographics) and variables available – can lead to biases ITT

Age > 65

Uplift Modeling• Statistical model, based on experimental

data• Predicts who the intervention is likely to

be most impactful for• Don’t need heuristics (other than

choosing inputs) – let data do the personalization for you

Robert Ellis


Eric Williams


Sean Duffy


Sami Lukens

spruced.

59

Subgroup Analysis: Physical Activity Engagement

• Physical activity is a large part of a healthy lifestyle• The second phase of the Omada Program focuses on

increasing patient’s physical activity:1. Providing pedometers to patients to collect physical activity (“steps”)

data2. Educational components and health-coach interaction about exercise3. Setting daily “step goals” for participants

60

Subgroup Analysis: Personalization of Step Goals

Challenge: can we provide personalized step goals to increase physical activity behaviors?

Patients are challenged with daily ‘step goals’ to increase physical activity

61

What does the data tell us:


62

Historical steps recorded, segmented by age/BMI

Assign each participant a personalized step goal based on: similar age/BMI historical mean + 20%


63

Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)

Personalized

Static (7500)

Step

s Tra

cked

/ W

eek

Program Week Program Week


64

Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)

Adaptive goals may be more impactful for younger [18-30] males

PersonalizedStatic (7500)

Step

s Tra

cked

/ W

eek



65

PersonalizedStatic (7500)

Step

s Tra

cked

/ W

eek


No apparent effect in middle-aged adults

Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)Subgroup Analysis: Personalization of Step Goals




Guided by heuristics (e.g. demographics) and variables available – can lead to biases ITT

Age > 65




choosing inputs) – let data do the personalization for you

Robert Ellis


Eric Williams


Sean Duffy


Sami Lukens

spruced.




Guided by heuristics (e.g. demographics) and variables available – limiting and can lead to biases ITT

Age > 65




choosing inputs) – let data do personalization for you

Robert Ellis


Eric Williams


Sean Duffy


Sami Lukens

spruced.

Experiment: Health coach feedback on participant food tracking

Context:• Building awareness of nutrition through food tracking is a

core part of the Prevent program• Ran experiment to understand the effects of coach feedback

on participant food tracking• Participants in the experiment received feedback about their

food choices from their coaches, those in the control arm did not.

Experiment: Providing impactful food feedback

Experimental design

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weigh gain – please provide feedback


1. Candidate Selection• Participants who track a meal & predicted to gain weight

2. Randomization• 50% are eligible to receive feedback

3. Prescription Generation• Coaches are notified of tracking behavior, with suggested

feedback4. Intervention Delivery

• Coach reaches out to participant with feedback on the flagged meal

5. Monitor outcomes

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weigh gain – please provide feedback







5. Monitor outcomes

Experimental design

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback



2. Randomization• 50% are eligible to receive feedback (other 50% in ‘control’)




5. Monitor outcomes

Experimental design






5. Monitor outcomes



Experimental design








5. Monitor outcomes

Experimental design

5. Monitor outcomes 10-15% increase in days with meals tracked


8-12% increase in meals with healthiness tracked5. Monitor outcomes


+8% in relative week-16 weight loss

5. Monitor outcomes


Using experimental data for personalizationUplift modeling, built on experimental data, can tell you probability of incremental response for an individual, given a hypothetical intervention

When delivering interventions:

• The Sure Things: response regardless of intervention• The Lost Causes: no response regardless of

intervention• The Sleeping Dogs: less likely to respond with

intervention• The Persuadables: only respond to because of

intervention

Uplift modeling targets The Persuadables and leaves the Sleeping Dogs alone.

Robert Ellis


Eric Williams


Sean Duffy


Sami Lukens

spruced.

Uplift modeling Example: Food Feedback

• Random forest uplift model trained on likelihood of participant continuing to track meals after receiving food feedback

• Model accounts for participant demographics, program behavior and response to previous feedback

• Preliminary results: targeting ~80% of eligible population with uplift model increases response ~2% (from 5% to 7%)

Outline

79

Omada Health• Context! Who are we? What do we do? Data science?





WHAT’S NEXT?

80

THANKS!@omadahealth

@sciencethedata@ryancquan

"using data science to design effective precision preventative behavioral medicine" - ryan...

Technology