building a data-driven culture
TRANSCRIPT
![Page 1: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/1.jpg)
May 3, 2023Presentation name 1
Building a Data-Driven CultureOlof HoverfältDirector, Sanoma Games September 2, 2015
![Page 2: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/2.jpg)
@Hoverfalt
![Page 3: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/3.jpg)
Sanoma Games is a diverse yet compact and lean unit
B2C & B2B, web & mobile Autonomous unit Key functions “in-house”
Business development
Marketing
Design
Development
Sales
![Page 4: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/4.jpg)
DESIGN
BUSINESS ANALYTICS
UX Business
![Page 5: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/5.jpg)
Being data-driven requires much more than the right tools
http://toolmonger.com/category/85/computer/page/2/
Handheld USB Snake
Scope
![Page 6: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/6.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 7: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/7.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 8: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/8.jpg)
http://theleanstartup.com/
![Page 9: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/9.jpg)
http://www.startupolic.com/launching-your-service-think-about-ux/; https://en.wikipedia.org/wiki/Lean_startup
An MVP is “a version
of a new product
which allows a team
to collect the
maximum amount of
validated learning
about customers with
the least effort"
The MVP approach is useful at all levels of development, from single product feature to new business concepts
Adopt MVP thinking in all development
![Page 10: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/10.jpg)
http://www.yesware.com/blog/how-to-close-more-deals-with-lean-email/
A continuous Build-Measure-Learn cycle enables fast learning but is really sensitive to friction in workflow and cooperation
Ensure all key functions are fully dedicated and the workflow is frictionless
![Page 11: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/11.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 12: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/12.jpg)
A data-driven culture cannot be created by coercion, but must be built through intrinsic motivation from its visible benefits
Transparency Autonomy Ownership
![Page 13: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/13.jpg)
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
Improvement actions
xxx
Revenue
ROS&packag. campaign vol.
Videocampaign vol.
Pricing
Discounts
Campaign volume
Optimization
Direct
Referral
SEO
Ret. visitors
Potential issues
Site problems
Ad blockers
New visitors
Reach
Monetization
Visits
Premiumfill rate
Premium eRPM
Remnant eCPM
Unique visitors
Media sales breakdown by contributing component
Impression success rate
nn%
nn%
nn% mm%
nn%
nn%
nn% lost
Ensure overall understanding of how the business works and of the contribution of and the dependencies between its components. Make deep data available
In order for a team to develop and fine tune an engine, everyone must know how it works
![Page 14: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/14.jpg)
Adoption of a data-driven approach requires for people to see its benefits; how they could kick ass even more at what they do
Image: http://blog.crazyegg.com/lessons/show-benefits/
Make the personal benefits of being data-driven visible.Show how its autonomous pursuit enhances mastery.
Super You!Data-drivenness
You
![Page 15: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/15.jpg)
Ownershipis not about doing what I tell you to do
is not about not doing what I tell you not to dois about doing what I don’t tell you to do
Ensure ownership of benefits, not deliverablesFocus on why, not what
![Page 16: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/16.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 17: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/17.jpg)
Data-drivenness, at both individual and team level, is a capability that can and should be developed
http://skepdic.com/tencommands.html
+Encourage autonomous reflection but ensure conceptual guidance
Reflection Key concepts
Being data-driven is not only about sensibly answering questions, but about finding them in the first place
It cannot itself be standardized into rules or processes, but it
can be guided by high-level key concepts
![Page 18: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/18.jpg)
A core principle of learning is to be hypothesis-driven. Random experimentation will yield various outcomes but little insight
Make continuous hypothesis-driven split testing the standard approach in all development activities across functions
Hypothesis:[A statement that can be shown true or false]
Design a test and a metric that confirms the hypothesis at a certain pre set limit
Test in a real setting with real usersPay attention also to
what is NOT happening
Confirm or reject the hypothesis based on
the chosen metric and pre set limit
Add the learnings to your body of
knowledge, also the invalidated hypotheses
Build on the knowledge to form the
next hypothesis
![Page 19: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/19.jpg)
Data without insight can be destructive as it may lead development the wrong way look like improvement
DAU(daily active users)
Downloads(i.e. not even activations)
Activity delta(change in user engagement)
Churn(users who quit)
>Vanity metricIndicator of
actual use
>Leading indicator of churn
ActionableLagging indicator
Not actionable
Put a lot of effort into designing KPIs (Key Performance Indicators) that are both sensible and actionable; such that steer development initiatives toward your ultimate business goal
![Page 20: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/20.jpg)
Main KPIs should be static but reflect the current growth phase, while tactical tools can be whatever helps improve the KPIs
High-level (growth-stage-specific) KPIs to continuously optimize
Tactical tools do explore user behavior and to measure impact from change on KPI(s)
http://lanesplitter.jalopnik.com/x-ray-artist-reveals-the-inner-beauty-of-classic-motorc-1698564514
![Page 21: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/21.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 22: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/22.jpg)
A simple tool that is actually used provides infinitely more value than a sophisticated one that is not
https://xplr.com/xplr-umbrella-dataviz-on-top-of-unsupervised-machine-learning/
>Make sure everyone has easy access to all tools they might need and know how to use them
![Page 23: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/23.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 24: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/24.jpg)
Goals set at function level can often be unaligned or conflicting, breaking joint effort and leading to sub-optimal performance
UX
Set high level common goals to ensure everyone works and cooperates to optimize the whole as opposed to sub-optimizing their own components
Business Manager goal:Visit eRPM
(effective revenue per session)
Product Manager goal:Avg. DAU(daily active users)
![Page 25: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/25.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 26: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/26.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 27: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/27.jpg)
Example:Continuous improvement and A/B testing
![Page 28: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/28.jpg)
A/B testing is about continuous and systematic gradual improvements that over time build increased performance
2.5x+ Improvement
![Page 29: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/29.jpg)
Seemingly small but sustained improvements can have a very large impact on overall performance
49% Improvementmin to max
25% Improvementmean to max
![Page 30: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/30.jpg)
![Page 31: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/31.jpg)
Example:Retention (how well you keep your users)
![Page 32: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/32.jpg)
![Page 33: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/33.jpg)
![Page 34: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/34.jpg)
6.1%Retention D1
![Page 35: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/35.jpg)
![Page 36: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/36.jpg)
![Page 37: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/37.jpg)
Count of users by level
![Page 38: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/38.jpg)
Count of users by level
![Page 39: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/39.jpg)
Count of users by level
![Page 40: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/40.jpg)
6.1%Retention D1
47%Retention D1
25%Retention D7
6.2%Retention D90
Compare with6.1% D1 Retention
in the previous example
![Page 41: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/41.jpg)
Example:LTV ((Customer) Life Time Value)
![Page 42: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/42.jpg)
User profitability is basically determined by LTV* and CAC*. Being able to even roughly predict LTV can be very insightful.KPIsDAU
Monthly rev.
Retention D1 43 %
Retention D7 24 %
Retention D14 19 %
Retention D30 12 %
Conversion
ARPDAU
ARPPDAU
LTV
CAC
ARPDAU nn € nn €
Extr. tot. days (≤120th) nn nn
LTV nn € nn €
* LTV = (Customer) Life Time Value ; CAC = Customer Acquisition Cost
![Page 43: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/43.jpg)
Example:Liigapörssi (KPIs and tools)
![Page 44: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/44.jpg)
![Page 45: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/45.jpg)
![Page 46: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/46.jpg)
Direct
Org. search
SEM
Display
Other
Our main time unit of comparable comparison is a game “phase”. The key desired user behavior varies within the phase
Free
Trades
1
Pool Free
Trades
2
Pool Free
Trades
6
Pool
Direct
Org. search
SEM
Display
Other
Pay(Revenue)
Join pool(Activation)
Sign up (Acquisition)
Play the whole phase(Retention)
Continue to next phase(Retention)
Sign up or reactivate
(Acquisition)
Desired user behavior
Create pool& invite
(Referral)
Invite friends(Referral)
Make trades(Activation)
Buy trades(Revenue)
Create team (Acquisition)
Phase 1 Phase 2 Phase 3-6Season
AcquisitionActivation
RetentionRevenue
Referral
Acquisition
ReferralRevenue
Retention
Activation Focus area
![Page 47: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/47.jpg)
We use various tools to identify development opportunities, run tests and measure impact on the main high-level KPIs set
Area KPI
Acquisition
Signed up (%)
Created team (%)
CAC (€)
ActivationActive user (%)
In pool (%)
RetentionIn-phase retention (%)
Phase change retention (%)
Revenue
LTV
ARPPU
Conversion
Referral Invited friend (%)
High-level (growth-stage-specific) KPIs to continuously optimize
Tactical tools do explore user behavior and to measure impact from change on KPI(s)
… …
Funnels
Emails
Anything!
Custom Dimensions using hashed user IDs enables user
level analysis, which is VERY powerful
![Page 48: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/48.jpg)
Some tools we are currently using
Tool Domain Sought benefits
Email Improve open rate and CTR
Email (transactional) Identify issues in use and funnels
Funnels Improve user onboarding and payments
Ecommerce Research revenue by product, user type, etc. (Enh. Ecom. Trac.)
Engagement Research anything at the user level!
Engagement Retention, User level progression
User Acquisition Volume and CPI (Cost Per Install)
Landing pages Optimize CTA (Call To Action) conversion
UI heat maps Research behavior and improve UI
KPI dashboard Communication and awareness
![Page 49: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/49.jpg)
Some suggested further reading
* http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version
Slideshare*
![Page 50: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/50.jpg)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
![Page 51: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/51.jpg)
@Hoverfalt
![Page 52: Building a data-driven culture](https://reader036.vdocuments.net/reader036/viewer/2022081414/5886b8501a28ab6d0e8b76b1/html5/thumbnails/52.jpg)
May 3, 2023Presentation name 52