Using online tools to help us assess our PLE work

Presented by Fiona MacCool, Your Legal Rights Project ManagerCLEO (Community Legal Education

Ontario / Éducation juridique communautaire Ontario)

October, 2013

What we’ll cover:• Introduction to Key Performance Indicators• Google Analytics (The Basics!)• Online User Testing• User Experience Surveys• Social Media Evaluation• Questions and Discussion

Let’s think about evaluation not as a single measurement in time nor as a vast collection of statistics you collect for funders. Evaluation, especially when it comes to online tools or websites, should be an ongoing process that ensures that continuous improvement is happening in ways that are measureable, evidence-based, and that contribute to the strategic goals of the organization.

(EASIER SAID THAN DONE!)

What are Key Performance Indicators?

Performance measures (or KPIs the terms are more often used interchangeably these days) are objective evidence of the degree to which a performance result is happening over time. They are feedback about your progress in improving aspects of your organization.

(Source: http://staceybarr.com)

Why use KPIs?• They allow us to set targets and improve our

site/projects over time• They allow us to take action to improve• They let us measure the viability of time consuming

activities – should we keep doing what we are doing?• They help us answer the question - What is the

point? Why are we doing this? Are we delivering on our promises?

Key Learning(s)… • This is not something you should be left to do on

your own as the “web person” or project manager. You should brainstorm every element of this process with others, including board and management• There is no one magic tool that can do this. You need to come at with combination of approaches and modify them regularly.• We need to share our strategies and techniques with one another as we are breaking new ground here.

The Process

1. Why - Look at your Vision Statement (for the org) – how can your website help you deliver on this objective (along with other channels)2. How - Strategic Plan, or Services Review Plans (one way our sector has advantages over the private sector) – Integrate key initiatives and where the org is heading to meet core goals and objectives3. Outcomes/Results - “Defining Success” – Looking at why and how – what outcomes or results will create the collective “fist bump” – what does victory/success look like for the org and for the project?

What makes a good KPI?“We know we will be successful when this RESULT is achieved”

Types of Success Evidence

1. Financial – online purchases, tickets sold, fundraising $ - Success Evidence – “Met fundraising goals”

2. Behavioural – Are people “doing” what you hoped they would (differently than without your site/project) - Success Evidence – “Increase in registrations for and repeat participation/viewings of webinars over time”

3. Experiential – Is the process better, are they able to complete their task more easily? – Success Evidence: Time required to submit an online request is reduced, client intake interview less time consuming

4. Attitudinal – Getting a new perspective on how things are done or what steps to take – Evidence of Success “Organizations perceive you to be collaborative and responsive to feedback”

5. Technical – slow site – needs to be faster - Evidence of Success – “Average page load time reduced” – Fewer “page not found” errors

Examples of KPI Metrics

• % of content greater than 1 year old – (insert appropriate time frame based on sector/topic)

• % of content voted up or down or rated• % Average # of edits prior to publish• % Average time to respond to email

questions/enquiries• % of users able to complete their task• % of users reporting satisfaction and/or that they

would recommend the site to others over time

Key learning(s)…• It is worth the time making a few customizations or

configurations to your website or online project early on to make sure are actually able to collect what you plan to measure . Google Analytics is great, but it’s probably not enough and nor is a simple user survey. Ask your developers about this early on if you can.

• Raw numbers of stats are less helpful and not easily actionable. It is better to establish a way to measure percentages and shifts in percentage patterns.

• Try to measure monthly averages or annual benchmarks – comparisons over time are key.

How to get started…

• Think about a project in your organization…• Think of what the goal of the project is?• Think of ONE result of that you would like to see

for the project.• Think of one piece of evidence you might use to

prove whether you have achieved this goal.• Draft that into a KPI (Key Performance Indicator)

Website SampleProject Goal Result Measurement

CLEONet.ca –a PLEI site for service providers was re-branded and launched as Your Legal Rights, a site aimed at the general public in Ontario

Expand site audience from service providers to also reach “General Public”

Within 1 year of the launch of YLR, we should see a 30 – 50 % increase in visits over the last year of CLEONet.ca

Within 1 year of launch we should see an increase in the % of users who report using the site to find information for a friend or family member rather than in their work or to help a client.

- Google Analytics Visits and Page Views

- User survey asking question of audience/purpose on CLEONet and then on YLR for key periods - % shifts in breakdown of purpose over time.

Project SampleProject Goal Result Measurement

PLEI Webinars for service providers in Ontario

Increase the reach of PLEI from local in-person workshops to wider online availability through YLR

Within 1 year of project launch we should deliver 10 -12 webinars reaching over 100 people live and over 1000 watching the archived versions.

Within 1 year of project launch, identify and partner with 4-6 organizations to develop and deliver webinars

- # of webinars conducted and archived to the site- # of live participants- # of views of archived webinars- Each of the metrics looked at as % from year to year

- # of organizations presenting webinars with CLEO

Surveys on whether webinars met user’s learning needs

Project SampleProject Goal Result Measurement

At Your Legal Rights we want to make sure we are responding to outreach emails in a timely and helpful manner

YLR should be an authoritative PLEI site for Ontario and must therefore provide information that is relevant.

% User survey respondents that agree/strongly agree that YLR’s response was relevant to their inquiry

Follow up survey sent to people who received a response from YLR’s outreach team. (Monthly)

Count of survey respondents that “Agree” or “Strongly Agree” that YLR’s response to their inquiry was relevant

Count of responses to the question vs. count of outreach emails sent

Trends examined over time

Benchmarking and BaselinesBenchmarking vs. Baseline

• Benchmarking – comparing one’s business process against competitors (industry standards – difficult in this sector to know)

• Baseline – initial known value which is used for comparison with later data (don’t forget to establish/retain stats before re-launching or redesigning your site)

Measurement – planning development processes testing

• Baseline from launch – For example: last 3 months of CLEONet to first three months of Yourlegalrights.ca, then year to year etc.

• Numbers of likes on your FB page or followers on Twitter over time• Target your baseline to your audience (service providers in Ontario,

general public) – Note that Google Analytics does Country and City only

Featured ToolsWhat are users doing on your site and how?

• Google Analytics – Tracks use and can follow “goals” or “events” set by you based on KPIs

• Heat maps – Crazyegg.com and Clicktale as well as some online usability testing tools show you who is clicking what, how quickly and how often

• Clicktale – records user sessions on web site – free account for 45 recorded sessions a month – shows you what/where users clicked – should help you see recurring problem areas

• Google Adwords – Great tool to track keywords and compare your page rankings and click-rates

Featured ToolsOnline User Testing

• Usability testing – You can pay third party sites/services to set up task-based tests your site with unknown people based on demographics you select, or you can do it yourself with tools like VerifyApp based on invitations to volunteers (cheaper than running them yourselves, especially if you serve a large Geographic area)

• A/B testing – Google’s A/B testing tool lets you split traffic in half so one half of users see one site/page and the others see a second version – reports will tell you that the new site reaches more goals – Try this with your email bulletins with MailChimp

Featured ToolsVoice of the User

• 4Q/IPerceptions – Survey that is used widely and asks 4 basic questions to random visitors specific to the purpose of their visit and their task completion and site satisfaction – This tool has a variety of monthly plans with add-on features

• Online Polls – asking a single question at key intervals let’s you get quick answers to questions about your users – Examples – Poll Daddy and Poll Everywhere

Bounce rate is the percentage of visits that go only one page before exiting a site: If the point of our work is to get people to find information quickly and leave our site(s) to get what they need, how can we know if a bounce is a success or whether they have lost interest or found the site useless/frustrating ?

Google Analytics – Basics

Custom Dashboards

Set widgets for the stats you care about most – In this case, visits, bounce rates, sources and goals

Use Google Analytics to Calculate Percentage Shifts Over Time

Set “Goals” in Google Analytics • If there are things you really want your users to “do” on your site, set them up as “Goals” in Google Analytics. Each time a user completes a Goal, a conversion is logged in your Google Analytics account.

• Give each goal a monetary value merely to rank them in relationship to one another. If a visit to a particular area of your site is more important/strategic than another give it a higher $ value. This allows you to answer the question “Which pages/sections are worth our time?”

• Improve a section of your site and see the Goal value rise and fall “at a glance”

Examples of Goals Types and Reports

Heat Maps and Screen Recordings

Google Adwords - Keywords

Adwords is a great tool to suggest keywords and help you know which words are commonly used in Ontario. It also tells you your site’s average position in Google search results.

User testing Best Practices• Should be task-based rather than “do you like

our site?”• Should not be “guided” or assisted in any way • Results should be observable and you only

need enough “testers” to identify trends or repeated problems

• Should be done early enough that you can make changes

In-Person User Testing

• Should be facilitated or led by a neutral 3rd party that can provide tasks and instructions and summarize findings

• Ideally, testers should verbalize their process, their confusion, their opinions as they go

• And again, ideally you should be able to watch and observe these results to make your final conclusions

Virtual or Online User Testing• Because we are trying to reach a vast

geographic area with limited funds, we elected to conduct our user testing for Your Legal Rights online in two phases

• Phase 1: Testing with wireframes – Design-free blue prints

• Phase 2: Re-testing with design mock-ups

Online User Testing: A Case Study• In April and then in July 2011 we conducted sets

of online usability tests for the Your Legal Rights website using a tool called VerifyApp.

• In April, we presented 15 volunteers with a set of test based on 3 – 6 wireframes. We then analyzed results and made revisions to our design plans. In July and then conducted a set of linked tests with 3 - 6 design mock ups including an option for users to annotate the page with comments or suggestions. For these later tests, we put out a wider call and had closer to 25 volunteers.

Wireframe Test: Where would you click to find information on eviction?

Mock-up Test: Where would you click to find information on eviction?

Sample Test Result Showing Heat MapWhere would you click to find information on Eviction?

Examples of Comments Reports

You can use these tools to test accessibility, navigation, emotional responses, user’s memory, and A/B testing

Samples here are from http://verifyapp.com/

A/B Testing• Create two “competing” versions of a page and

then have Google create a test where 50% of users see one and 50% see the other and then examine results. Which page had a higher bounce rate? Which page led to the next level you were hoping users would find?

• Create two “competing” designs of your email newsletter or e-bulletin and see which one has a higher open or click-through rate. (MailChimp)

User Experience Surveys

• Create a post-visit survey and set an invitation rate (20 to 50% of visitors) for key time periods throughout the year.

• Ask users what their primary task was, whether they completed it, what their “next step” is, and if they would recommend your site. They can also leave comments

IPerceptions (formerly 4Q)

Rate of return based on Invitation RateWe set the Invitation rate to 20% for desktop, 10% for mobile and 20% for tablet and leave it in place for 4-6 months each year to compare and benchmark success

User Experience Survey

Comments by Purpose of VisitChallenge: People keep “anonymously” telling me what they wish they had found and I can’t help them find it!

Anecdotal Observation: Now that our site is not for service providers, users can be more negative

Social Media – Followers/UsersTwitter

As of July 31, 2012 we have sent out 3900 tweets and have 1,560 followers.

As of March 31, 2012 we have sent out 5,035 Tweets to 1,998 Followers

This represents a 28.07% change in the # of followers to @legalrightsON on Twitter.

Facebook

As of the end of July 2012, the Your Legal Rights Facebook page had 600 fans or “likes”.

As of the end of March 2013, the Your Legal Rights Facebook page had 2,031 fans or “likes”.

This represents a 238.49% increase.

Social Media– Measuring EngagementTwitter - “Re-tweets” and “Mentions” – Hard to measure this over time, better to log it on a schedule for benchmarking or baseline purposes. Lots of tools give you a “glance” of the last 90 days but for our sector, we need longer timeframes.

Facebook - Facebook Insights Reports for Pages shows “Impressions” which refers to the number of people who saw your postings as well as user engagement in terms of the number of people who clicked on any of your postings or created stories based on content posted to your Facebook page.

Social Media – Facebook Insights

Social Media – Klout Score

Tweetreach

https://tweetreach.com/reports/7661919?v=2

Discussion Questions? How do we figure out who the audience is and why they

are visiting our sites or using our PLEI? It is more than “purpose” and can’t really be captured in demographic data…

How do we know what success really looks like without any “industry benchmarks” or with such goals so difficult to measure because of client privacy?

When our sites cover vast ranges of topics and given the gaps in what legal info is needed, what insights can we bring to the reasons certain topics or keywords are most visited/searched?

Let’s do a sample together…• Think about a project in your organization…• Think of what the goal of the project is?• Think of ONE result of that you would like to see

for the project.• Think of one piece of evidence you might use to

prove whether you have achieved this goal.• Draft that into a KPI (Key Performance Indicator)

Keep in Touch!Fiona MacCool

Your Legal Rights Project ManagerTel: 416-408-4420 ext. 829

Email: fionamaccool@yourlegalrights.on.caWeb site: http://www.yourlegalrights.on.ca

Twitter Feed: http://twitter.com/legalrightsONFacebook: http://www.facebook.com/yourlegalrights