New thinking on

benchmarking customer experience

Vicki Howe, Head of

Product Development

April 2015

2

© HouseMark 2017

Contents

1 Key points............................................................................................................................ 3

2 Introduction ........................................................................................................................ 4

2a Satisfaction benchmarking from HouseMark ............................................................... 4

3 Star benchmarking and its benefits ............................................................................. 5

3a Latest sector-wide results from HouseMark’s Star benchmarking ...................... 5

3b Key drivers of satisfaction ................................................................................................... 6

4 Star, tracking and transactional surveys and the need to review ........................ 7

4a The consultation ..................................................................................................................... 8

4b What the sector said ............................................................................................................. 8

5 StarT: Our way forward for transactional surveys ................................................. 10

5a Star developments in response to the consultation ................................................ 11

Star and StarT, and the intellectual property rights in them, are owned by HouseMark Ltd (“HouseMark”). We give

express permission for anyone who is a social landlord, or their subcontractor, to use, store and transmit Star and

StarT. This includes the right to copy the Star and StarT questions and the approach set out in written guidance

provided by HouseMark. Star and StarT surveys and questionnaires do not have to be acknowledged to

HouseMark. The Star logo (available at www.housemark.co.uk/star) may be reproduced on questionnaires, reports

and other materials based on Star on the basis that you use it in the same way that HouseMark uses it, including

without limitation, its format, colours and proportions. You are expressly prohibited from using the Star logo in a

way that is detrimental to the reputation of HouseMark or Star or in a way that is not in keeping with the way that

HouseMark uses the logo.

You acknowledge that Star and StarT has not been developed to meet your individual requirements. Whilst care

has been taken to ensure the accuracy and completeness of the questions, we exclude all conditions, warranties,

representations or other terms, so far as is permitted by law, concerning Star and StarT, including without

limitation that relating to fitness for purpose, accuracy, completeness or intellectual property. We hereby disclaim

all liability and responsibility for your use of Star and StarT, or any part thereof, so far as it is permitted by law.

3

© HouseMark 2017

1 Key points

HouseMark owns and manages the Star satisfaction framework for the social housing

sector. Using this framework, over 350 UK social housing landlords regularly collect and

benchmark perception satisfaction survey data.

Landlords use satisfaction data, of which Star is a part of, for their own performance

monitoring and to inform business decisions.

Taken as a whole, the HouseMark Star satisfaction dataset reveals overall sector trends

and typical drivers of satisfaction. Our latest data shows overall satisfaction levels have

remained constant on average for general needs and housing for older people tenants.

Since Star’s launch in 2011, a wider range of survey methods have become popular with

landlords than is covered by the Star framework.

Transactional surveys, such as repairs, lettings or ASB surveys, are not covered directly

by the Star framework. However, some HouseMark members wanted to explore the

appetite for guidance and a benchmarking framework specifically for transactional

surveys.

HouseMark, in association with market research company Acuity, conducted a major

consultation exercise with HouseMark members in late 2014 / early 2015 to find out

more about the extent to which transactional surveys are utilised, current expectations

for customer satisfaction benchmarks and the continuing role of Star.

We found that Star remains popular and continues to have an important role to play, with

91% telling us that Star still has a place.

The evidence base arising from this consultation exercise has led to the development of

StarT by HouseMark – a transactional satisfaction survey framework that is

complementary to, but separate from, Star.

HouseMark is delivering new and refreshed guidance and benchmarking services to help

our member compare satisfaction for both perception (Star) and transactional (StarT)

surveys.

4

© HouseMark 2017

2 Introduction

2a Satisfaction benchmarking from HouseMark

In 2011, HouseMark launched the Star (Survey of Tenants and Residents)

framework - a periodic perception-based survey that provides social housing

landlords with the means to compare satisfaction results with each other. Today,

nearly four years later, around 350 landlords regularly collect and benchmark this

information using HouseMark services.

Landlords say they continue to find Star an invaluable resource for understanding

how tenants and residents feel about the services they provide and they use this

information to inform business planning and resource allocation. As a wider range

of survey methods has become available for measuring and understanding

satisfaction, there has been a growing appetite to combine this over-time

perception survey data with real-time customer feedback. Many landlords are now

carrying out transactional surveys triggered by specific events or interactions with

customers; for example following a responsive repair, complaint, or letting.

The purpose of this report is three-fold. Firstly, it provides a recap on the benefits

of Star, and what HouseMark’s Star dataset tells us about the latest trends in

satisfaction.

Secondly, it outlines the results of our research into:

• The extent to which transactional surveys are being used, and are evolving,

in the sector

• The demand for benchmarking data gathered from transactional surveys

• What guidance the sector might need on transactional surveys and

understanding their relationship to perception customer satisfaction

surveys (such as Star surveys).

And, thirdly, to introduce StarT; our new framework for benchmarking transactional

survey data.

Used together, the Star and StarT frameworks will provide a fuller picture of

comparable customer satisfaction than has ever before been available.

5

© HouseMark 2017

3 Star benchmarking and its benefits

HouseMark developed Star for the social housing sector. Our website1 contains

details about how to run a Star survey together with a full set of questions covering

the core areas of activity; as well as several optional and model questions for

specific landlord services.

This standardised approach to satisfaction measurement enables landlords to

make meaningful performance comparisons with other landlords working in the

social housing sector and monitor trends over time.

Satisfaction measurement provides a valuable source of data for measuring and

reporting tenant and resident-focused outcomes and engaging tenants and

residents in what their landlord does. Most landlords will carry out a variety of

surveys to capture satisfaction data. Star is not designed to meet all the surveying

needs of a typical landlord; however, it will deliver a robust, central foundation for

capturing tenants’ and residents’ general perceptions about their landlord.

The framework is designed with sufficient built-in flexibility to ensure that landlords

are able to measure what they need for Star compliance in ways that best suit their

organisational requirements and those of their tenants and residents.

Satisfaction measurement is also about gathering robust, actionable data to inform

business decisions on changes to service delivery. Without subsequent action,

satisfaction measurement has no purpose; without robustness any action may be

misdirected.

For landlords in Scotland, HouseMark teamed up with SHBVN (Scottish Housing

Best Value Network) to produced Star in Scotland which combines the regulatory

requirements under the Scottish Social Housing Charter with Star's established

framework for comparing satisfaction.

Once a Star (or Star in Scotland) survey has been conducted and collated, the

results can be entered into HouseMark’s Star module. Star data, when combined

with cost and performance data in HouseMark’s core benchmarking service,

provides the basis upon which to judge value for money and identify the best

opportunities for improving performance.

3a Latest sector-wide results from HouseMark’s Star

benchmarking

HouseMark’s annual report on Star benchmarking data based on surveys carried

out in 2013/14 found that overall satisfaction has remained constant for

leaseholders, general needs and housing for older people tenants between

2012/13 and 2013/14. There has been a small increase (1% point) in overall

satisfaction for supported housing tenants and a larger increase (8% points) for

shared owners.2

1 https://www.housemark.co.uk/subscriber-tools/benchmarking/survey-of-tenants-and-residents 2 Comparisons used a balanced panel of landlords that submitted performance data for both years, and excluded

those which submitted data from the same survey two years running.

6

© HouseMark 2017

As in previous years, this year’s annual report found that tenants are more satisfied

with the service provided by their landlord than home owners (leaseholders and

shared owners). In 2013/14, general needs tenants were less satisfied than those

living in housing for older people, and shared owners are more satisfied overall

than leaseholders.

When the results are broken down by region, they show that satisfaction for

London-based organisations is around 10 percentage points lower than other

regions across all quartiles. Outside London, the scores are remarkably consistent,

with only a few points separating the regions at each quartile threshold.

Although less up-to-date than the data captured by HouseMark’s Star

benchmarking3, the CLG’s English Housing Survey publications4 also offer some

supplementary insight into recent trends in satisfaction. The 2012/13 survey

findings show that the general trend in satisfaction with accommodation,

neighbourhood and repairs among social housing tenants over the five years

between, 2008/09 and 2012/13, is very slightly upwards.

3b Key drivers of satisfaction

HouseMark’s annual Star benchmarking report uses correlation and regression

analysis to assess how much each benchmarked Star question influences overall

satisfaction. This demonstrates that every core measure5 is significantly correlated

with overall satisfaction, and that this correlation is positive i.e. as one core

measure increases in value, so does overall satisfaction.

Repairs and maintenance has the biggest influence on overall satisfaction, as it did

in in 2012/13 and 2011/12. Landlords whose residents are satisfied in this area

tend to have the highest rates of overall satisfaction too. The next biggest

influencer is listening and acting to views, closely followed by the Value for Money

(VFM) of rent and the quality of home. This implies that getting resident

involvement right is an important factor in achieving good overall satisfaction

3 The English Housing Survey’s breakdowns by tenure (including social housing tenants) for its 2013/14 survey are

not published until July 2015. 4 https://www.gov.uk/government/collections/english-housing-survey 5 The seven core Star satisfaction measures cover overall satisfaction, the home, neighbourhood, VFM rent, VFM

service charge, repairs and listens/acts on views

7

© HouseMark 2017

scores, as well as VFM and the quality of home, as lower ratings in these areas will

have a negative impact on overall satisfaction.

4 Star, tracking and transactional surveys and the

need to review

Since the launch of Star, a much wider range of survey methods has become

popular with landlords for measuring and understanding customer satisfaction.

Notwithstanding this broader range of survey options, 91% of our consultation

respondents are clear that they want to continue deploying Star as part of their mix

of approaches.

A Star survey is a perception survey aimed at measuring all6 customers’ views,

impressions and opinions about their landlord and the services it delivers.

Historically, perception surveys were carried out at specific points in time, typically

every two or three years. Today, however, many landlords carry out their Star

perception surveys as tracking surveys which means running the same survey at

set intervals throughout the year, such as weekly, monthly or quarterly.

We define transactional surveys as those that collect customer feedback about an

interaction or event, gathering data linked to a recent service while the interaction

is still fresh in customers’ minds. The distribution of a transactional survey to a

respondent (e.g. by way of posting a survey or making a phone call) is triggered by

an event such as a responsive repair, a complaint or a letting. These surveys are an

invaluable way of measuring the customer experience and views on service quality

at the operational level, obtaining feedback very quickly after the event.

The distinction between the two survey types of perception and transactional is

not the actual questions asked, but the trigger for distributing the survey. For one

type, the survey distribution follows an interaction or an event (transactional)

whereas the other type (perception) does not.

HouseMark recognises the need to be even clearer in our benchmarking guidance

about what is and is not a Star survey, and specifically how transactional surveys

relate (or not) to STAR.

The purpose of our review was therefore, to understand the extent to which

transactional surveys are used in the sector and to assess the demand for the

benchmarking of customer satisfaction data from transactional surveys. The

review also sought to understand what guidance the sector might require on

transactional surveys and their relationship to perception customer satisfaction

surveys (such as Star surveys), and to develop a new framework for the use of both

surveys to successfully capture robust and representative data.

6 To capture ‘all’ views, the landlord will either conduct a census survey or a carefully constructed sample survey to

ensure statistically valid representation

8

© HouseMark 2017

4a The consultation

The consultation set out to understand the extent to which transactional surveys

are used in the sector and to assess the demand for the benchmarking of

customer satisfaction data from transactional surveys. It also sought to understand

what guidance the sector might require on transactional surveys and their

relationship to perception customer satisfaction surveys. We also took the

opportunity to seek feedback on the current Star survey framework.

HouseMark began the project in October 2014 with a roundtable discussion to set

the scope of the review. Participants were invited from a cross-section of

HouseMark members, some of whom were critics of Star. Following the event, two

online member surveys and ten in-depth telephone interviews were carried out in

December 2014 and January 2015. HouseMark members were also asked to

submit examples of their Star and transactional surveys.

The online survey was aimed at research and policy staff, directors and managers

of specific service areas. It was split into two parts: Part A focused on

benchmarking and the role of transactional and Star surveys; Part B asked further

in-depth questions about the particular transactional surveys conducted by the

landlord. Part A of the survey was completed by 178 staff from 166 landlords, with

a further 73 landlords continuing to complete the more in-depth Part B. A charitable

donation was made for each survey response received. In addition to the online

consultation responses, a total of 341 examples of transactional and Star

questionnaires were sent to HouseMark.

In-depth structured telephone interviews were conducted with senior executives in

the social housing sector. These interviews captured their opinions on the use

made of customer satisfaction information in improving, developing and tailoring

services to meet customer requirements, monitoring operational performance and

contract management and establishing a sector-wide approach to benchmarking

transactional surveys.

Detailed findings from the review are available for HouseMark members on our

website.

4b What the sector said

The review confirmed the popularity of Star surveys with 91% of respondents

confirming that they use the framework and plan to keep on doing so.

Virtually all landlords carry out transactional surveys (99%). Almost all of the

landlords conduct responsive repairs surveys (94%), with a high percentage

carrying out ASB complaints (84%), new lettings (76%), and complaints (77%)

surveys.

Telephone (85%) and postal surveys (83%) are the most popular research method

used by social landlords to survey residents; but many landlords also use online

(56%), face-to-face (51%), mystery shoppers (48%) and focus groups (47%). Two

of the challenges facing many landlords are falling response rates for postal

surveys and the often disappointing levels of take up for online surveys. Although

many landlords use social media to communicate with customers, few are

9

© HouseMark 2017

conducting customer research over these newer platforms.

Of the landlords who conduct Star surveys, the majority continue to carry out one-

off surveys (66%), but a third now use tracking surveys (33%). Postal surveys

remain the most popular survey method for Star surveys (68% compared to 25%

telephone), but a third of those carrying out postal surveys are considering

changing survey methods. This desire to investigate alternatives could be related

to the concern about falling response rates for postal surveys. Those adopting the

second most popular method of telephone surveys are much more likely to retain

this method in future – only 4% would consider a change.

We asked about the extent to which landlords include the seven core questions in

their Star surveys. Six of the seven core Star questions are included by well over

90% of landlords (94% to 100% depending on the question); only the question on

whether service charges provide value for money, which is not applicable for all

landlords, is less frequently used (81%).

In our discussions with HouseMark members prior to the consultation, we had

picked up a limited amount of concern about two specific core Star questions and

we sought opinions on their future in this consultation. We found the majority of

landlords favour keeping the question on whether rent provides value for money

(66% find it useful). Just over half of the landlords who responded wanted to keep

the question about landlords listening and acting on residents’ views (53%).

In order to compare satisfaction results, the requirement for all participants in a

benchmarking exercise to work to the same answer scale for a particular question

is as important as the wording of that question being the same. For all core Star

questions, a five-point descriptive text scale of very satisfied, fairly satisfied,

neither, fairly dissatisfied, very dissatisfied was adopted when the framework was

set up in 2011. We asked about response scales again in our latest consultation. A

majority of respondents (53%) favoured retaining a five-point scale (which includes

the middle ground option ‘neither satisfied nor dissatisfied’). The use of descriptive

text for the answer scale (rather than a numeric scale) remains the most popular

option (78%).

Two-thirds of landlords would like to see a set of survey questions developed for

transactional surveys by HouseMark (65%), with a higher number wanting to see

guidance and advice from HouseMark (as ‘thought leaders’) on survey methods and

sample sizes for transactional surveys (87%).

Landlords want to see more customer satisfaction measures added or included in

HouseMark’s benchmarking modules and a link to comparable data sets available

outside the sector. The consultation found a strong demand for guidance, sharing

of good practice and learning events for the sector.

10

© HouseMark 2017

5 StarT: Our way forward for transactional surveys

In response to the feedback from the

consultation, we are developing StarT

(Survey for Tenants and Residents

Transactions), our new benchmarking

service for transactional surveys.

Just as Star provides a comparable

framework for a landlord’s main tenant

and resident perception satisfaction

survey, StarT will provide a framework for

comparable transactional surveys.

The Star and StarT frameworks are

distinct and separate, but together they

provide a fuller picture of landlords’

customer satisfaction than Star was able

to deliver alone.

The feedback from our consultation confirms our belief that when it comes to

comparing satisfaction results, Star (perception) data and StarT (transactional) data

are also distinct and separate. We will not be mixing data from the two different

types of survey in our benchmarking services.

Our new StarT guidance on the use of transactional surveys is available to

HouseMark members. It includes a set of questions for transactional surveys as

well as providing advice on survey methods, sample sizes and other areas

highlighted in the review.

Through the introduction of StarT, we are expanding our benchmarking services to

include clearly defined transactional customer satisfaction measures, starting with

the most popular surveys identified in the review – repairs, lettings, ASB and

complaints.

During 2015/16, landlords who wish to take up this offer of new transactional

satisfaction benchmarking services for repairs, lettings, ASB and complaints will be

able to adapt their survey methods and questions to meet our new StarT

framework. From April 2016, the new online benchmarking systems will be available

to HouseMark subscribers to input their survey data and see how they compare.

We have developed standard StarT questions for each of these four transactional

survey areas. There is built-in flexibility and landlords do not have to ask all of the

questions in each service area in order to participate in our benchmarking.

Question sets may be expanded from other sources such as optional StarT

questions and your own questions to tailor the survey to meet your exact needs.

The choice of StarT questions was strongly influenced by the information obtained

during our consultation from the on-line questionnaire respondents, in-depth

interviews and hundreds of examples of customer survey

questionnaires/telephone scripts.

11

© HouseMark 2017

StarT will enable more landlords to take part in comparable survey measurement

appropriate to the survey type, and a wider range of measures to be included in

HouseMark satisfaction benchmarking.

5a Star developments in response to the consultation

Our priority has been to respond to the consultation feedback in relation to

providing StarT guidance and benchmarking solutions for transactional

satisfaction surveys for those HouseMark members who want this.

We also learned from the consultation that, while there is very widespread (91%)

support for the Star framework, there are a number of areas where landlords would

like further guidance and there were various suggestions for development of the

Star framework.

During the spring and summer of 2015, we will be considering and consulting

further on:

• The future form of the ‘listens and act’ core question in Star

• The introduction of social value and/or ONS well-being measures into the

Star optional question set

• How to address some concerns about whether the ‘neither’ responses are

included or excluded in the calculation of satisfaction results

• Expanding the range of filters available in on-line benchmarking of both Star

and StarT for those landlords who want to narrow their peer group of

comparison to surveys and organisations that are most like their own.

The overall timeline for HouseMark delivery on both a refreshed Star and a new

StarT framework to our members is:

This ‘new thinking’ report on benchmarking

customer experience 30 April 2015

Detailed findings from our consultation 30 April 2015

StarT – a guide to running transactional

surveys, including StarT questions 30 April 2015

Upgrade of the key Star documents:

• Star features

• A guide to running Star

• Star questions

July 2015

Introduction of on-line StarT benchmarking

services April 2016

Upgrade of on-line Star benchmarking

services, including more filter options 2018

4 Riley Court, Millburn Hill Road, University of Warwick Science Park, CV4 7HP.

Company Reg. No 3822761 VAT Reg. No. 744 0353 53

T: 024 7646 0500

E: info@housemark.co.uk

W: www.housemark.co.uk