analytics academy 2017 presentation slides
TRANSCRIPT
![Page 1: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/1.jpg)
WELCOME TO ANALYTICS ACADEMY!
![Page 2: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/2.jpg)
A/B Testing and Creating a Culture of Experimentation
![Page 3: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/3.jpg)
![Page 4: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/4.jpg)
A/B experiments show users different versions of your site and then compare results
![Page 5: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/5.jpg)
TEST FIRST: FAST
● Can often mock up a feature in the testing tool first, without involving a tech queue
● Measuring results isn’t affected by seasonality, or other marketing efforts, or changes to the consumer mood because you are testing one group randomly divided, so all these factors are controlled for
● Results are statistically tested and validated
BUILD FIRST: SLOW
● Higher up-front investment: Have to invest in building the feature without knowing if it will work
● Hard to measure results: you roll out the feature and compare conversion before (5-7% for last two months) to after (5.5-6.7% for month after). Is this an improvement or normal variation? Is it affected by seasonality? Did the email campaign that went out last week affect this rate?
Benefits of A/B testing
![Page 6: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/6.jpg)
Experiment 1: Tools in StoreHypothesis: ● Replacing "Tools" with a more
learning-centered phrase will produce more click throughs
What’s the result?
![Page 7: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/7.jpg)
Experiment 1: Tools in StoreHypothesis: ● Replacing "Tools" with a more
learning-centered phrase will produce more click throughs
What’s the result?● Learning Tools click rate up 16%● Teach Yourself click rate up 27%
![Page 8: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/8.jpg)
![Page 9: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/9.jpg)
Hypothesis: ● Changing to one year would have
no negative effect on subscription conversion
What’s the result?●
What we think●
Experiment 2: One year vs 10 issues
![Page 10: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/10.jpg)
Hypothesis: ● Changing to one year would have
no negative effect on subscription conversion
What’s the result?● One year variation subscription
rate up 9%
What we think● People understand the value of a
year more than number of issues
Experiment 2: One year vs 10 issues
![Page 11: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/11.jpg)
![Page 12: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/12.jpg)
Experiment 3: Formatting on product pageHypothesis: ● Improving formatting would
increase product purchase
What’s the result?●
What we think●
![Page 13: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/13.jpg)
Experiment 3: Formatting on product pageHypothesis: ● Improving formatting would
increase product purchase
What’s the result?● No statistical difference in
product purchase
What we think● Was formatted description too
long? Should we have short text and preview page?
![Page 14: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/14.jpg)
![Page 15: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/15.jpg)
A/B testing can inspire cultural change● Practice with A/B tests builds experimentation muscles
○ People practice the steps to build a good experiment so they start to feel obvious
○ A/B tests require good methodology: you are forced to pick a goal to measure; you automatically have a control group; the software collects and reports on the results
○ The benefits of the speed/clarity from these experiments increase demand for similar speed/clarity in areas outside the website
Ideal state for all business stakeholders for all questions: always ask, “Can this be an experiment?”
![Page 16: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/16.jpg)
Basic human nature makes this hard
● Short term, it feels easier to make a decision based on gut feel, or defer to highest paid person’s opinion (HiPPO), or just try something and see what happens without formalizing a hypothesis or measuring the result (but still call it an experiment)
What makes it hard to experiment?
![Page 17: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/17.jpg)
In the long run, it is actually a LOT easier to run an experiment
● Fail fast: have an idea? Experiment with a minimum viable product to see if the idea deserves further development--or not
● Decisions vs more discussion: The organization can move a lot faster when there’s certainty around a course of action. When there’s uncertainty, healthy discussion can sometimes sour into multiple meetings and prolonged debates, or, perhaps worse, unspoken doubts sap the momentum for the group moving forward
Experimentation done well becomes self-reinforcing, as people see how much easier/faster they can work
A/B testing shows experiments are easier
![Page 18: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/18.jpg)
● New product development: set out hypotheses about the market (e.g. “managers want to buy HBP materials to help their direct reports”) and then test with customers (e.g. customer interviews where we learn that there’s an equally large market from coaches). Key is to build and test in stages, so you validate hypotheses along the way
● Email testing: split list as randomly as possible and send different emails to the two groups
● Before and after testing: Create an insider newsletter and compare subscriber engagement before and after
● A/B testing: use a formal A/B testing platform on the website
Some ways that HBR experiments
![Page 19: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/19.jpg)
Change is as good as a rest: Sometimes a change tests well just because it is a change and gets people’s attention. Change a button from red to blue, you may get higher click throughs; effect diminishes over time, then six months later change it back to red and get higher click throughs
Focus on the big picture: don’t look at a change in isolation; look at the total impact. Adding a newsletter widget that gets clicks is good, but does it increase the total newsletter signups (or just cannibalize the clicks you are getting from other widgets)? Does the additional visual clutter lower overall engagement (higher bounce, lower time on site)?
Follow some best practices
![Page 20: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/20.jpg)
But remember we are not a labHaving a culture of experimentation does not mean that your group transforms into a medical lab where we need 98% certainty and huge sample sizes to make a decision
Perfect is the enemy of the good: It’s better to have 20 good experiments than 3 perfect ones
Hurdle: is the experimental results better information that what you would have used otherwise? (e.g. better than gut instinct?)
![Page 21: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/21.jpg)
Questions?
![Page 22: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/22.jpg)
ResourcesGoogle analytics experiments: FREE! (but anecdotally pretty hard to use)
Optimizely: easy interface, great training resources to help you get acquainted with testing (we started here)
VWO: may be cheaper than Optimizely
Adobe Target: more robust, integrates with Adobe Analytics in a very powerful way (we moved here in January)
![Page 23: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/23.jpg)
Facebook Ads:Audiences and ImpactJoseph Casciano, Harvard Public Affairs and Communications
![Page 24: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/24.jpg)
How much to spend?
Why are we spending it?
![Page 25: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/25.jpg)
● Low-value objective● Larger audience
● High-value objective● Smaller audience
![Page 26: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/26.jpg)
![Page 27: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/27.jpg)
Problems with Interest Targeting
● Often inaccurate, so serves your ad to the wrong people.
● Hard to get a precise, smaller, well-targeted audience.
● To big audiences, Facebook serves your ad to whoever is cheap and easy.
![Page 28: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/28.jpg)
Business Manager
↓
Assets
↓
Audiences
↓
Create Audience
↓
Custom Audience
![Page 29: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/29.jpg)
![Page 30: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/30.jpg)
![Page 31: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/31.jpg)
Metric Time
![Page 32: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/32.jpg)
Relevance Score
![Page 33: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/33.jpg)
CPM (cost per 1,000 impressions)
![Page 34: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/34.jpg)
Results and Reach
![Page 35: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/35.jpg)
Custom audiences make metrics matter
● Don’t provide contextless numbers about faceless masses.
● Tell concrete, true stories about your valued audience.
● So: “We reached half of our email subscribers on Facebook, half of who watched the video for more than three seconds.”
● Not: “We reached 132,674 Malaysian bots who couldn’t even theoretically fly into Cambridge for our symposium.”
![Page 36: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/36.jpg)
● Use custom audiences○ To guide your budget○ To make metrics matter
![Page 38: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/38.jpg)
Google Analytics Tips Tricks
Elizabeth Brady, EWB Analytics
![Page 39: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/39.jpg)
IntroElizabeth Brady, Founder & Principal Web Analyst - EWB Analytics LLC - launched
March 2010
Specialties: Google Analytics and Google Tag Manager implementations, site audits,
web analytics support during site re-launch, measurement strategy and ongoing
analysis
Harvard groups I have collaborated with since 2012: Digital Communications, Harvard
Alumni, Harvard Admissions, Harvard Innovation Lab, Kennedy School, Harvard
Library, Harvard Learning Portal, HWPI, Ash Center
Contact: [email protected]
![Page 40: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/40.jpg)
Keep It Clean
![Page 41: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/41.jpg)
Filters Can Help PreventInternal Traffic
m 1
Maintain ‘exclude’ filter of
known internal IP
addresses
Ghost Spam
Maintain ‘include’ filter of
valid site hostnames
Crawler Spam
Maintain ‘exclude’ filter of
list of known spam
referrers
![Page 42: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/42.jpg)
Filter: Exclude Internal Traffic by IP AddressInternal traffic inflates conversions & conversion rates. Check
current IP address by visiting whatismyip.com
IP AddressesUse regular expressions for a range of IP
addresses (ask IT for office IP range)
Dynamic IP Addresses
(residential)
Verify/update regularly
Test ActivityUse custom dimensions to track test users
even when not on internal network
![Page 43: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/43.jpg)
Filter: Test Accounts by Custom Dimension● Use a custom dimension to
identify a test visitor who
visits a specific internal page
(webadmin, test, etc)
● Set the custom dimension at
the ‘user’ level
● Create a filter for any traffic
with that custom dimension
value
![Page 44: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/44.jpg)
Ghost referrer spam:● Never actually visits your site● Sends data via the ‘measurement profile’
randomly to your GA account (became an issue only with Universal Analytics)
● Sends data with a missing (not set) or inaccurate hostname
● Can be prevented with a valid ‘hostname’ (include) filter
Prevent Ghost Referrer Spam
![Page 45: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/45.jpg)
Filter: Valid Hostname(s)
![Page 46: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/46.jpg)
Crawler spam:● Actually crawls/visits your site so the traffic appears legitimate● Filter this traffic by filtering on ‘campaign source’● Sample ‘exclude’ filter for known spam crawlers and domains
referenced as referrals from spam crawlers:semalt|anticrawler|best-seo-offer|best-seo-solution|buttons-for-website|buttons-for-your-website|7makemoneyonline|-musicas*-grat
is|kambasoft|savetubevideo|ranksonic|medispainstitute|offers.bycontext|100dollars-seo|sitevaluation|dailyrank
● Full set of 4 filters for crawler spam can be found here:http://help.analyticsedge.com/spam-filter/definitive-guide-to-removing-google-analytics-spam/
Filter: Crawler Spam
![Page 47: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/47.jpg)
Language Spam - New Spam in 2016Language Spam
● Rather than referrers, the spamming
sites inserted spam messages into the
‘language’ reports
● Most do not use valid hostnames so
this would also be prevented with a
‘hostname’ include filter
● Additional exclude filters can be added
to address language spam
![Page 48: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/48.jpg)
Lowercase Filters● Especially when starting a new
view, lowercase filters can avoid
capitalization inconsistencies
● Recommended for - page,
campaign
(medium/source/campaign),
search term (on site search)
![Page 49: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/49.jpg)
Query String Cleanup● Google Analytics includes any query string
parameters (after the ‘?’ as part of the URL)
● Leads to multiple versions of the same ‘page’
and a challenge aggregating data
● Parameters to exclude can be identified in a
list in the view settings, or you could ‘go
nuclear’ and exclude them all with this filter
on the right
● Full URL with query strings (or just query
strings) can be captured as a custom
dimension to be viewed when needed
![Page 50: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/50.jpg)
Check Your Setup
![Page 51: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/51.jpg)
Referrer Exclusion List (Property Level)● Make sure your site subdomain/s is
included (new properties set up with
Universal Analytics will have this set
up on creation but any older site rolled
over to Universal Analytics did not
automatically have this configured)
● Include any off-site flows (login
validation, back-end sites like
pin1.harvard.edu) to prevent triggering
a new session
● Do not set up harvard.edu in the
exclusion list (that will prevent any
other Harvard sites showing up as
‘referrals’) - they will be ‘direct traffic’
![Page 52: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/52.jpg)
Check ‘Exclude Bots & Spiders’ (View)● Excludes traffic from sites on the IAB
(Interactive Advertising Bureau) list of
known bots & spiders
● Sometimes these can be contracted
services like site response time (like
Gomez) that execute javascript and
would otherwise show up in reports
● It is recommended to leave this
unchecked for the unfiltered view
![Page 53: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/53.jpg)
Link Google Search Console (Property) for Organic Search Trends
● Google Analytics no longer has much
insight into Google organic search
keywords
● Link your site’s Search Console
(formerly ‘Webmaster Tools’) account
for impressions/clicks on Google
![Page 54: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/54.jpg)
Remember Campaign ‘Timeout’ (Property) is Configurable
● Standard campaign setting is 6 months
(sessions and conversions will be
credited to the last campaign in the
past 6 months)
● This explains why you may see
traffic/conversion for ‘old’ campaigns
● Your business group may decide you
need a shorter or longer campaign
timeout
![Page 55: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/55.jpg)
Data Import (File Upload) Can Extend Analysis
● Data import lets you append
data to any dimension (standard
or custom) you collect
● The actual import can be a
simple text file upload
● Some uses might be to add
details around campaigns, add
authors or other details to
content pages, or group
information differently than
they way it is grouped in Google
Analytics
![Page 56: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/56.jpg)
Reporting & Analysis
![Page 57: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/57.jpg)
Custom Reports
Don’t dig for your data!
404s
Social Media Details
Top Pages by Type
Top Events by Type
Deep dive into a certain source of
traffic (ex: email campaigns)
![Page 58: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/58.jpg)
Tracking 404’s
● No extra tagging needed
● Report Filter: Page title contains ‘Page Not Found’
● Dimensions: URL (page requested), Previous Page (might be entrance),
source/medium (more important for entrance pages to understand source of
traffic)
![Page 59: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/59.jpg)
404 Error Report
● Monitor 404 volume over time
● Monitor broken links
● Set up 301 redirects where needed
● This report is very helpful after a site re-launch
![Page 60: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/60.jpg)
Social Media Details
● Report Filter: Channel = Social
● Dimensions: Medium, Social Network (or Source)
● ‘Social’ = tagged social campaigns, ‘referral’ = organic social traffic (no camppaign
tags)
![Page 61: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/61.jpg)
Pages by Type
● Report Filter: Page contains <URL identifier for type of content>
● Dimensions: Page
Possible content: blogs, story pages, article pages, FAQ’s
![Page 62: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/62.jpg)
Events by Type
● Report Filter: Event category = ______________
● Dimensions: Event label, event action
Possible events: document downloads, offsite links, navigation links, carousel clicks
![Page 63: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/63.jpg)
‘Unique’ Metrics - Pageviews● To report the number of sessions that
viewed a page, use ‘unique pageviews’
● GOTCHA - do NOT combine page
with sessions as a custom report (GA
WILL let you set this up, but sessions
are ONLY associated with the entry
page)
![Page 64: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/64.jpg)
‘Unique’ Metrics - Events● To report the number of sessions that recorded a certain event, use ‘unique
dimension combinations’
● This shows the sessions with the event for whatever dimension combination is
presented in the report
![Page 65: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/65.jpg)
Custom Segments● Create a custom segment to filter any report by sessions/users meeting specific
criteria
● Some common segments include:
○ Sessions from a specific campaign
○ Sessions that viewed a specific page
○ Sessions that registered a specific ‘event’
● Then apply a segment to a basic or custom report, for example:
○ Geographic reports
○ Traffic (source/medium) reporting
○ Technology: device/browser/OS reporting
![Page 66: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/66.jpg)
Custom Segment - Viewed the homepage●
![Page 67: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/67.jpg)
Custom Channel Groupings● CUSTOM channel groupings give you
the flexibility to roll up the data the way
you want to see it
● They are retroactive, but are specific to
the user account where they are created
but can be shared like other assets
● For the Gazette, we break out
Harvard.edu referrals, other Harvard
referrals, and non-Harvard referrals as
separate ‘Channels’
![Page 68: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/68.jpg)
Custom Channel Groupings● CUSTOM channel groupings are a
view-level setting - be sure to find the
‘custom’ groupings rather than the
‘channel groupings’ (changes to the core
channel groupings will not be retroactive
and will only impact data collection
moving forward)
![Page 69: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/69.jpg)
Helpful Toolkit
![Page 70: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/70.jpg)
Google Tag AssistantChrome Extension
● Quickly check the status of Google
Analytics and Tag Manager code on
any page
● Red/yellow warnings identify tagging
problems
![Page 71: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/71.jpg)
EditThisCookieChrome Extension
● View cookies set on a site
● Delete selected, or all, cookies on the site
without having to clear all of your cookies
for other sites
![Page 72: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/72.jpg)
Google Data StudioGoogle’s new dashboard Tool
now offers free, unlimited
dashboards, with great
integration with Google
Analytics and other Google
products
Features: interactive filters,
flexible formatting, multi-page
dashboards
datastudio.google.com
![Page 73: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/73.jpg)
Takeaways - Top Tip From Each Topic!1. Filters - maintain data integrity by collecting the cleanest data you can in your
production a view (a non-filtered view should also exist), slides 4-12.
2. Settings - key settings to check include referral exclusions (include your own
harvard SUBdomain/s) and make sure bots/spiders are excluded.
3. Reporting - remember to use ‘unique’ metrics when reporting the number of
sessions with a specific page/event.
4. Tools - download ‘Google Tag Assistant’ for very user-friendly feedback on tag
set-up and data collection.
[email protected] - Feel free to reach out with specific questions!
![Page 74: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/74.jpg)
Copyright © President & Fellows of Harvard College
Responding to Analytics with SEOMarcus Dandurand - March 30th, 2017
![Page 75: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/75.jpg)
How to Measure Traffic from Search EnginesIn Google Analytics: Source/Medium = Google/OrganicIn Adobe Analytics: Marketing Channel = Natural Search
2
![Page 76: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/76.jpg)
Are we getting enough Search traffic?
3
![Page 77: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/77.jpg)
Optimization is never done!
4
➢ Benchmark against yourself
➢ Compare traffic Year-Over-Year
➢ Try optimizing existing pages
➢ Create new pages to target new keywords
➢ Think beyond “branded” keywords
![Page 78: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/78.jpg)
Search Traffic Year-Over-Year
5
![Page 79: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/79.jpg)
Wait… What?!Traffic is down!!! Is it something we did? Did Google change its algorithm?Will it fix itself?What can we do?
6
![Page 80: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/80.jpg)
First, a few SEO myths...
7
1. We don’t know what Google wants
2. The algorithm changes too often
3. SEO is an attempt to “game the system”
4. SEO is a job for IT
5. My CMS has SEO built-in
![Page 81: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/81.jpg)
What do search algorithms care about?
8
Relevance
Performance
Authority
![Page 82: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/82.jpg)
What is page “Relevance”?Your page content closely matches a keyword search phrase.
9
![Page 83: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/83.jpg)
A few ways to improve a page’s relevance:
10
➢ Keyword Research - Relabel content using “outside voice”
➢ Breakup Content - Each important idea should have its own landing page
➢ Accurately describe all page components∙ Page Titles/Meta data∙ Navigation links∙ Section Headers∙ Etc.
![Page 84: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/84.jpg)
What is page “Performance”?Pages are useful, Pages load fast, Site is accessible to humans & robots.
11
![Page 85: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/85.jpg)
A few ways to improve a page’s performance:
12
➢ Reduce page load time
➢ Make sites mobile friendly
➢ Improve click-through rate in search engine results
➢ Make sure websites can be crawled/indexed properly by search engines
![Page 86: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/86.jpg)
What is page “Authority”?Every link acts as an endorsement of a page’s credibility.
Both External and Internal links!
13
![Page 87: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/87.jpg)
Authority began as Google “PageRank”
14
![Page 88: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/88.jpg)
A few ways to improve a page’s authority:
15
➢ Create resources that people will share (inbound linking)
➢ Use 301 redirects (site cleanup)
➢ Restructure website navigation to distribute authority to your most important pages
Distributing Authority:Are the important links on your page 1/10 or 1/100?
![Page 89: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/89.jpg)
Working Knowledge Website Updates:SEO lessons learned the hard way.
16
![Page 90: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/90.jpg)
Mid August 2015
17
![Page 91: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/91.jpg)
Working Knowledge has a search traffic problem!
18
![Page 92: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/92.jpg)
Working Knowledge search traffic improves!
19
![Page 93: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/93.jpg)
Working Knowledge search traffic improves!
20
➢ Sept – Nov 2015: down 43% vs. Prior Year
➢ Sept – Nov 2016: up 46% vs. Prior Year
![Page 94: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/94.jpg)
So, what did we fix?
21
![Page 95: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/95.jpg)
1) New dynamic landing pages
22
➢ Hundreds of Topic landing pages were not indexed by Google. New browse page was behaving like a single dynamic page. (Performance)
➢ Each page had the same Title & Meta Description (Relevance)
![Page 96: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/96.jpg)
2) Deleted the Working Knowledge Archive
23
➢ Several articles were still very popular for search traffic. (Relevance)
➢ Many pages were still cited and linked to by important sources. (Authority)
Without proper redirects, the authority passed back to the home page was lost.
![Page 97: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/97.jpg)
Other optimization for Working Knowledge
24
➢ Created customized Titles & Descriptions for each page in the CMS. (Relevance)
![Page 98: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/98.jpg)
Other optimization for Working Knowledge
25
➢ Created new “display descriptions” visible on-page. (Relevance)
![Page 99: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/99.jpg)
Are You Making A Major Website Update?Please consider the following...
26
![Page 100: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/100.jpg)
#1) Navigation Links Transfer Page Authority
27
➢ Primary navigation create backlinks from every page on your site.
➢ Try to put your important pages in your primary navigation (but only if it makes sense).
➢ Try to remove links that are nice to have, but not critical.
➢ Adding new links will dilute the authority of existing links.
![Page 101: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/101.jpg)
#2) All URL Changes Need 301 Redirects
28
➢ 404 errors create a bad user experience and waste page authority.
➢ 302 redirects are “temporary,” so Google keeps the old page in the index. No authority is passed!
➢ 301 redirects are “permanent,” so the authority of old pages are passed.
![Page 102: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/102.jpg)
#3) Avoid Duplicate Content
29
➢ Each page should have a unique Title & Description.
➢ You should not be able to see the same page via two different URLs.
![Page 103: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/103.jpg)
Pop Quiz:
30
Which of the following URLs below are exactly the same as: www.hbs.edu/mba
A. www.mba.hbs.eduB. hbs.edu/mbaC. http://www.hbs.edu/mbaD. https://www.hbs.edu/mbaE. All of the above
![Page 104: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/104.jpg)
Free Tools for SEO Analytics
31
![Page 105: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/105.jpg)
Free Tools for SEO Analytics
32
➢ MozBar (Browser Extension) ● Page Authority, Inbound links
➢ SiteImprove - Free through HUIT!● Missing Page Titles, Descriptions, Broken Links, 302 redirects
➢ Google Search Console - Formerly “Google Webmaster” ● 404 errors, Organic Keywords, XML sitemap, Mobile issues
➢ Link Redirect Trace (Browser Extension) ● Follow the path of multiple redirects
![Page 106: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/106.jpg)
Thank You!
33
![Page 107: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/107.jpg)
From Big Data to Insights in Massive Open Online CoursesA Traveler’s Guide
Daniel SeatonHarvard UniversitySr. Research ScientistVPAL Research Team
![Page 108: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/108.jpg)
[email protected] Academy - March, 2017
• Consortium of Institutions creating MOOCs
• Maintain Open-Source Platform• Host Courses/Content• Lead Outreach• Maintain “https://www.edx.org”
• Partners from Higher Ed / Industry / Government / High Schools
• Create Courses/Content• Manage Courses in Open Online
(MOOC) and On-Campus (residential) settings.
• Perform Research into Teaching and Learning
Consortium Members
What is edX?
![Page 109: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/109.jpg)
[email protected] Academy - March, 2017
Data from Dec. 2016
http://harvardx.harvard.edu/Harvard University’s MOOC Organization:
• Partners with faculty to create open online courses
• Supports initiatives to use MOOC content beyond open online models
![Page 110: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/110.jpg)
[email protected] Academy - March, 2017
Data from Dec. 2016
Harvard University’s MOOC Organization:
• Partners with faculty to create open online courses
• Supports initiatives to use MOOC content beyond open online models
http://harvardx.harvard.edu/
![Page 111: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/111.jpg)
[email protected] Academy - March, 2017
Research Timeline and Perspective
2012
What are learners doing?
Who and where are our learners?
Besides open online, how else can we use MOOC platforms and content?
Why are learners taking courses?
2013 2014 2015 2016
Single MOOC
Transforming Advanced Placement High School Classrooms Through
Teacher-Led MOOC ModelsSeaton, Hansen, Goff, Houck, Sellers
Many MOOCsContext around
MOOC enrollments
Alternative MOOC Models
![Page 113: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/113.jpg)
[email protected] Academy - March, 2017
• 6.002x: Circuits and Electronics (first MOOC from MITx - now edX)
• Over 100K enrollees• Over 7K certified users• Over 100GB of data from clickstream• Limited profile information
• MITx now a member of edX: ~ 100 open access courses
![Page 114: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/114.jpg)
[email protected] Academy - March, 2017
We started with “what” people are doing in 6.002x
Transition between resources
Nodes are resources(size ~ time spent)
Edges are transitions (size ~ weight)
Who does what in a Massive Open Online Course?Seaton, Bergner, Mitros, Chuang, Pritchard ( Comm. of the ACM - 2014)
Analyzed learner interactions with all aspects of 6.002x. Particular focus on time-on-task and resource-use during problem solving.
Measurements• Time-on-Task• Resource Interactions• Daily/Weekly Progress• Transitions between resources
during problem solving
![Page 116: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/116.jpg)
[email protected] Academy - March, 2017
• HarvardX and MITx Working Paper #1
• Now had access to all course data from MITx and HarvardX
• Addressed “what” people were doing, and “who” they are, across 17 MITx and HarvardX courses
Key Takeaways:
1. Courses are very different.
2. Registrant diversity is immense compared to residential.
3. Participation greatly varies.
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
![Page 117: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/117.jpg)
[email protected] Academy - March, 2017
What are learners doing across MITx and HarvardX?
% G
rade
% Chapters Accessed0
100
100
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
![Page 118: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/118.jpg)
[email protected] Academy - March, 2017
What are learners doing across MITx and HarvardX?
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
![Page 119: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/119.jpg)
[email protected] Academy - March, 2017
Cross course surveys launched in 2014 addressing broad issues across MITx, but teaching experience was central issue.
Results from 11 spring 2014 MITx MOOCs:
• 28.0% (9451) self-identify as past or present teachers (navy).
• 8.7% (2847) current teachers (orange).
• 5.9% (1871) teach/taught the topic (gray).
On average across courses, ~ 8% (1 in 12) of comments are from current teachers.
For teachers that teach/taught the topic, the average across courses is ~6% (1 in 16).
PercentCommentsin Forum
Did not take
survey
Surveyed
SurveyedTeachers
Non-Teachers
43.8%
22.4%
33.8%
![Page 120: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/120.jpg)
[email protected] Academy - March, 2017
Why are we still not talking about course structure/design?+
Visualizing Course Design
![Page 121: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/121.jpg)
[email protected] Academy - March, 2017
Goals and Motivations
• Support HarvardX by collecting relevant stats on course structure.
• From a research perspective, identify canonical patterns in course development and better understand how those patterns affect behavior and outcomes.
Human beings, viewed as behaving systems, are quite simple. The apparent complexity of our behavior over time is largely a reflection of the complexity of the environment in which we find ourselves.
- Herbert Simon, “The Science of the Artificial”
Practical Motivation
Abstract Motivation
http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research
![Page 122: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/122.jpg)
[email protected] Academy - March, 2017
Visualizing Course Design
Key point: Use these visualizations to look across courses.
![Page 123: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/123.jpg)
[email protected] Academy - March, 2017
![Page 124: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/124.jpg)
[email protected] Academy - March, 2017
http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research
![Page 126: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/126.jpg)
[email protected] Academy - March, 2017
• Boston Public School students• Take MOOC online during school• Commute to BU weekly for
labs/recitations with TAs/Faculty
Of 34 regular and charter schools serving 16,165 students, 2 high schools offer algebra based AP® Physics 1. Only 60 BPS students took the AP® Physics 1 exam during the 2014-2015 school year.
BU Project Accelerate
• Open-online and teacher-led/flipped • All content open on edx.org• Special instances for teachers to
use content in classrooms• Showed 0.08 added to AP exam score per
hour usage above class average
http://vpal.harvard.edu/blog/complementary-models-mooc-instruction-advanced-placement%C2%AE-high-school-courses
![Page 127: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/127.jpg)
[email protected] Academy - March, 2017
Davidson Next - AP content for use by teachers and students
Program at Davidson College:
• Supplemental content for 14 Challenging Concepts in each AP subject.
• Challenging concepts determined using College Board exam data from 2011 to 2013. Piloted with Charlotte-Mecklenburg School System in 2014-2015 school year.
• Modules designed for each concept meant to facilitate use both in classrooms, and open online. Real AP Teachers from developed content with Davidson faculty.
• Courses released on edX.org and through a new Custom Course tool (CCX).
![Page 128: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/128.jpg)
[email protected] Academy - March, 2017
Bubble Charts for Detecting Daily Activity• Made these to help monitor teacher use of Davidson Next in Charlotte High
Schools
• Full time assessment coordinator worked with teachers on implementation and efficacy of content (collected district data and AP exam scores).
Transforming Advanced Placement High School Classrooms Through Teacher-Led MOOC Models
Seaton, Hansen, Goff, Houck, Sellers (MIT LINC Conference - May 2016)
Pilot program in North Carolina High Schools
Massive Open Online Courses via edX.org
Exam score residuals are then correlated with student usage relative to class median indicating 0.08 points per hour spent (p<0.05).
![Page 130: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/130.jpg)
[email protected] Academy - March, 2017
Open Source Tools for edX Data
• Harvard and MIT already share resources and code for analytics
• https://github.com/mitodl/edx2bigquery
• https://github.com/mitodl/xanalytics
• Open-Source Repos• Python + Google
BigQuery for aggregation of edX data.
• Dashboard via Google App Engine
![Page 131: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/131.jpg)
[email protected] Academy - March, 2017
edX Data Workshop Summer 2016
Meeting of data analysts and engineers in institutional roles responsible for edX data; 16 attendees from 11 institutions.
Goals for meeting:
• Discuss broader aspects of data sharing and analytics.
• Standup the Harvard/MIT edX Data Pipeline.
• Happy to report that each participant completed this task.
• Next workshop summer 2017?
• Hoping to broadly release workshop documentation in the spring.
http://news.harvard.edu/gazette/story/2016/07/moocs-ahead/
![Page 133: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/133.jpg)
[email protected] Academy - March, 2017
Many collaborators to thank before dicussion!
HarvardXAndrew Ho, Dan Levy, Jim Waldo,
John Hansen, Sergiy Nesterko, Justin Reich, Tommy Mullaney
Miki Goyal, Gabe Mulley,Carlos Rocha, Victor Schnayder,
Olga Stroilova, Brian Wilson
Julie Goff, Aaron Houck, Kristen Eshleman, Pat Sellers, Noelle Smith
Yoav Bergner, Cody Coleman, Isaac Chuang, Curtis Northcutt,David Pritchard, Saif Rayyan
VPAL Research TeamAndrew Ang, Glenn Lopez, Brooke Pulitzer, Yigal Rosen, Dustin Tingley, Selen Turkay, Jacob Whitehill, Joseph Williams
Lydia Snover, Jon Daries, Mark Hansen
Research and
Analytics
![Page 134: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/134.jpg)
Ditch the spreadsheet and tell the story
Katie HammerOffice for Sustainability and Harvard Public Affairs and Communications
![Page 135: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/135.jpg)
![Page 136: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/136.jpg)
How do you get your team to understand your analytics story?
![Page 137: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/137.jpg)
![Page 138: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/138.jpg)
![Page 139: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/139.jpg)
MATERIALS AND CONTENT CREATED:
GHG Landing page (OFS)
4 Page Climate Report PDF
Community-wide message from President Faust
Wide-format Gazette Story and Graphics
8 #HarvardClimateStories instagram profiles
Video targeted at social media
Custom social graphics for 12 Schools + departments
harvard.edu/climate modules
![Page 140: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/140.jpg)
Community-wide email sent by President Faust
52% open rate867 clicks
Gazette Story in the Daily Gazette 22% open rate637 clicks
Social Promotion (Harvard & OFS)
● Video with paid boost● Twitter & Twitter Moment● #HarvardClimateStories
Instagram Campaign
● 108,000+ video views● Almost all Schools shared
news w/ graphics● 17,694 likes; 84 comments
Inclusion in OFS December email newsletter
25% open rate507 clicks
Feature on Harvard.edu 3,010 clicks
External Press (Crimson, Harvard Magazine Story,NY Times, Boston Globe, WGBH etc.)
DISTRIBUTION EFFORTS:
![Page 141: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/141.jpg)
OFS Goal Page: 335
Gazette Story: 210
HUCE Site: 301
Clicks
![Page 142: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/142.jpg)
Try something new?Flaunt it.
![Page 143: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/143.jpg)
FACEBOOK AD CAMPAIGN: OFS
OFS Ad Spend: $170Duration: December 8 - 12
Audience: Targeted students and alumni (where Harvard was listed as School and age was 18+); OFS email list; People who liked our Facebook page
Reach (number of people that saw the post):● 42,019 total people reached● 16,624 people reached as a result of paid● 17,000 total video views; ● Cost per 1,000 people reached $10.23● $.05 per 10 second video view
Engagement (reactions, comments, shares):● 7,696 total actions● $0.27 per engagement ● 23 link clicks● Post generated 55 new GreenHarvard
Facebook page likes
Context:● Typical GreenHarvard video is viewed ~500
times
Note: Many comments did include mention of divestment; however about half were positive and congratulatory.
![Page 144: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/144.jpg)
Qualitative data matters too.
![Page 145: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/145.jpg)
TWITTER STRATEGY:
● Worked with Facility teams to create custom graphics optimized for social for Schools to use
● Partnered w/ HPAC to share on the @Harvard accounts● Outreach in advance to all digital counterparts at Schools/Depts● Created a Twitter Moment to capture various influencer
and School tweets about announcement
RESULTS:
● Initial tweet: Retweeted 66 times; Liked 102 times, Clicked 34 times● Moment tweet: Retweeted 33 times, Liked 92 times, Clicked 90 times● Retweets and original tweets from internal “influencers” like HBS,
HSPH, HAA ● Almost all 12 Schools promoted us in some way, in addition to the
Museums, Libraries, and various departments
![Page 146: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/146.jpg)
And they all livedhappily ever after...
![Page 147: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/147.jpg)
Lessons learned and opportunities
● Targeted outreach to Schools and Depts works; Schools/Depts find easier to promote when can link data/anecdotes back to them (social graphics received well).
● While School/Dept outreach worked and we did have some external influencer tweets (Climate Registry, USGBC), we should develop a more solid plan for faculty and social influencers in the future.
● #HarvardClimateStories campaign a success; 8 profiles in a month was ambitious; for future campaigns could start earlier and conduct interviews and shoots further in advance.
● Important to consider different outreach methods for different audiences; for example the social video was extremely brief but gave an external audience the message they needed “Harvard set an ambitious goal and they met it.”
● We should consider allocating time more evenly across a wide range of projects, considering goals, audiences, and reach (PDFs, videos, social campaigns). For example, though PDF a considerable amount of our time, the social/web reach was minimal.
![Page 148: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/148.jpg)
vs.
![Page 149: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/149.jpg)
Keys to a telling a good story
FormatChoose a vehicle that’s relatable (even if that means powerpoint). Keep it simple.
StyleUse language that seems right for your story (and for your client).
SettingSet your story by bringing in context to explain the why. Remember you control this!
ThemesLet the themes of your data shine by weaving them throughout your story.
IllustrationsImages, examples, and visual cues only add to your story.
ConclusionEnd your story with lessons learned and opportunities that leave the reader ready for your next story!
![Page 150: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/150.jpg)
![Page 151: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/151.jpg)
Thanks!Any questions ?◉ [email protected]
![Page 152: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/152.jpg)
Strategize, Synthesize, and JAZZERCIZE®
your Analytics Dashboards and Reports
with your host Aaron David Baker
![Page 153: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/153.jpg)
Remember Jazzercize®?
Problem:
Non-dancers are taking Jazz Dance classes because it is a great workout but aren’t interested in all the work on form and technique. They just wanna have fun while exercising.
Solution:
Create a fun jazz dance-style fitness class that’s interesting and fun!
![Page 154: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/154.jpg)
The power of FUN
![Page 155: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/155.jpg)
The problem was neither exercise nor jazz dance class
Exercise is a chore; we make it fun to get it done.
![Page 156: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/156.jpg)
It’s the same with creating/consuming analytics data
![Page 157: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/157.jpg)
Give them what they want!
The people who consume your reports want them to be engaging.
They don’t always have to contain charts and graphs.
Good document design also conveys professionalism.
Don’t just report on the content you produce or see, share screenshot highlights to give context.
![Page 158: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/158.jpg)
Great reports are persuasive and can change attitudes
![Page 159: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/159.jpg)
Introducing Scoop
HPAC’s analytics dashboard, built around public APIs to Google Analytics, Facebook, Silverpop, and more to come
Presents consistent, up-to-date performance data per story, post, and mailing
Makes comparative metrics possible with benchmarking and visualizations
![Page 160: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/160.jpg)
Introducing Scoop
We have a Strategy
● Identify and capture metrics that matter in one convenient place
We have Synthesis
● Stats from many different platforms in one place for easy reporting
We are working on that Jazz
● Make it beautiful and fun
![Page 161: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/161.jpg)
Strategize
![Page 162: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/162.jpg)
![Page 163: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/163.jpg)
The official Harvard Style Guidelines & Best Practices site has an updated Analytics with resources and setup information.
harvard.edu/guidelines
Use these best practices to ensure your site is up-to-date with the latest analytics code and tracking practices.
First, check that everything is in order
![Page 164: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/164.jpg)
Second, define key stats (these are just some examples)
Health stats are these
● Users/Sessions/Pageviews ● % New Users over time● Page speed● 404s
Strategic metrics are these
● Content performance (pageviews)● Content engagement
○ Time on page○ Scroll depth
● Users by○ New/returning○ Geolocation○ Content sections they visit○ Frequency of visits
● Content dimensions○ Content category/section/tag○ Content length
● Acquisition paths○ Search keywords
![Page 165: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/165.jpg)
Choose goals and metrics that make sense for your organization!
![Page 166: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/166.jpg)
Report on what’s exceptional and on what’s important
What’s exceptional
● Top content in terms of pageviews and/or engagement metrics (time and scroll)
● Large number of people reached or high number of impressions
● Social activity (likes, comments, shares, retweets)
● Unusual spikes in traffic* or unknown sources
What’s important
● Key initiatives○ President Faust’s priorities○ Special Gazette features
● Things you spent money on○ Paid social, AdWords, etc.○ Extra money gives you extra
metrics● Experiments
○ A/B testing ○ SEO
● Conversions
![Page 167: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/167.jpg)
Here we’ve chosen to highlight pageviews by channel and accumulated pageviews over time
● Helps us visualize content distribution and sources of traffic.
● Benchmarking is our own comparative metric—average daily pageviews of all stories in Scoop.
Scoop Example
![Page 168: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/168.jpg)
Synthesize
![Page 169: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/169.jpg)
The chore of reporting
● Go here● Find stats● Copy/paste into doc● Go there● Find stats● Copy/paste into doc● …
![Page 170: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/170.jpg)
Wouldn’t it be nice if the stats gathered themselves?
![Page 171: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/171.jpg)
Scoop pulls in stats from
● Google Analytics● Facebook● Silverpop
With plans to incorporate
● Twitter● Instagram● YouTube● Etc.
Anything with an API can be consumed.
Scoop Example
![Page 172: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/172.jpg)
![Page 173: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/173.jpg)
Synthesis isn’t just about automation
Automation is nice and does save a lot of time.
Linking things by a common element (like URL or topic) can make finding the stats easier.
Synthesis is really about telling the whole story.
Automated reporting cannot speak for you.
Talk about the why in your reports.
![Page 174: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/174.jpg)
My Weekly Report
Ultimate synthesis of what happened last week
Mostly highlighting what’s exceptional
Occasionally mentioning what is bizarre
Always as interactive as possible
● links go to actual online posts or to Scoop itself
![Page 175: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/175.jpg)
![Page 176: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/176.jpg)
Jazzercise!
![Page 177: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/177.jpg)
Back to Jazzercize®
Our reports are working, and people come to us for information, but how can we analytics reports more enjoyable?
![Page 178: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/178.jpg)
Design and data visualizations:
● Beautiful, clean, contemporary, and inviting design
● Display visually our data’s trends, patterns, and correlations
● Provide content creators and distributors with intuitive, at-a-glance insights about performance of published work
● Be designed with interactive development in mind
The plan for Scoop
![Page 179: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/179.jpg)
Your turn to talk:
How do you add jazz to your reports?
![Page 180: Analytics Academy 2017 Presentation Slides](https://reader036.vdocuments.net/reader036/viewer/2022070520/58f9c226760da32f4b8b6458/html5/thumbnails/180.jpg)
Thanks y’all!