howwillyoucustomise$ the$tracker? - jisc digital...

How will you customise the Tracker?

Overview 1


Overview This document helps you decide how you will customise the Student Digital Experience Tracker to ensure you are asking the right questions. You may want to share and discuss this document with other stakeholders. Decisions about how you customise the survey could make a big difference to the data you collect and what you can do with it, so we advise that you read through this document before you start. There is a separate 'How To' guide to using BOS that takes you through the process of customisation once you are ready to go ahead.

Looking at your unique survey, you will see that pages 1 to 6 contain the core Tracker questions and cannot be changed. Page 7 is where you have a chance to decide on any additional questions your learners will see and respond to. Page 8 collects demographic information and the information you will use to group your learners for comparative analysis. The questions on both these pages should be customised to suit your needs.

Page 7: ‘Your views’ There are five questions on this page (four in the Online version). You will see that all but the last question can be DELETED but not edited (with one minor exception). We have written and tested the content of these questions for you. These optional questions cover:

• Digital skills

• Digital safety and wellbeing

• Use of the VLE (or, for the Online Tracker, use of the online learning environment)

• e-‐Assessment (not included in the Online Tracker)

You and your stakeholders must decide which of these issues are priorities for you and your learners, and so which questions will be kept in the survey. You will DELETE the other questions. The one exception to the ‘no


Jisc Student/Learner Digital Experience Tracker http://bit.ly/1WJkMPS 2

edit’ rule is that if you use the question about the virtual/online learning environment you will be able to type in the name of your local VLE or online platform so that your learners will recognise it.

The last question can be EDITED OR DELETED. Like the others on this page, this question asks learners how far they agree or disagree with a set of statements about their learning experience. Unlike the others, you can write your own statements – though we suggest you limit these to no more than five. You and your stakeholders must decide whether you want to use this question template, and if so what question statements you will include. Bear in mind that a badly worded or over-‐complicated question at this stage in the survey will cause learners to drop out. So please use this option only if you are confident that you can design good statements and that you have time to pilot them.

If you do decide to write your own question statements for learners to respond to, look through the rest of the survey carefully to be sure that your customised question is similar in style. Notice that the core questions follow these important rules:

• Use clear, learner-‐facing language. Avoid jargon.

• Keep question statements short. If you need to clarify or exemplify, use the ‘Advanced’ tab to add ‘More information’ so that learners can click and read it if they want to.

• Ask about only one idea per statement, otherwise you will slow respondents down as they think about what you mean, and it will be harder to interpret your findings.

• Word statements positively so it is clear what it means to ‘agree’ or ‘disagree’. This does not mean your statements have to be positive about digital issues. Our question 8 asks respondents to agree with statements that have negative connotations, such as ‘When digital technology is used on my course… I am more easily distracted’. But it is clear that ‘agree’ means ‘yes, I am more easily distracted’. Note that if the statement had been worded negatively e.g. ‘I don’t concentrate as well’, there would have been room for confusion.

• Avoid leading statements and loaded language, which point respondents towards a preferred answer.

• Test each statement to ensure that learners understand it in the way you intended. Aim to have four or more learners work through the whole question, and have them think aloud as they do so. Eliminate or rephrase any statements that cause confusion or take longer to process. You could also talk to these learners more generally about the value of the questions you are asking. For example, are these issues important to them, and do they believe that any feedback they give will make a difference?

The format of the question gives you scope to ask about several different issues, and so explore questions within the Tracker that might otherwise require you to run a separate survey or consultation. Other questions on this page group themed statements together under a shared heading. You may want to follow this by giving your question a themed heading, or you could call it ‘Issues at [your organisation]’, or simply delete the heading entirely and ask about a range of different issues. For reasons of length and balance we ask you to use no more than five question statements in total, and if they refer to very different issues, please consider using fewer.



Important considerations

1. You do not have to use any of the questions on page 7. The core question set deals with the issues we find to be most important to learners.

2. If you do decide to ask optional questions, then for reasons of length, balance and participant commitment, there should be no more than three questions on page 7. So your page can be made up of: 0, 1, 2 or 3 optional questions OR 0, 1 or 2 optional questions plus the customised question. The customised question, if used, should have no more than five question statements.

3. When learners access this page they will not see any distinction between optional and customised questions, or between these and our core questions.

4. You may be interested to compare learners’ feedback over several iterations of the Tracker. If this is the case then you will want this page to stay the same in future years, and you need to choose issues that will continue to be important.

5. On the other hand, you may want to canvas learner views about an issue that is very current. Changing some of the customised statements every year means that the Tracker stays fresh and relevant. But bear in mind that if you ask about an immediate concern, learners will expect a rapid response to their feedback.

6. Questions on this page cannot be benchmarked within BOS. However, where enough institutions select to use an optional question, we will include this data in our synthesis report. So once we have published it you should be able to compare your findings with the sector as a whole.

7. This is the first time we have allowed users to write their own questions and we are interested to know how you find it. In future we could offer more customisation options, so that (for example) you could include a larger number of questions in the Tracker that you might otherwise ask in a separate survey. We will ask what you think about this idea as part of our evaluation.

Page 8: ‘About you’ There are four questions on this page (five in the Online Tracker) that you can use to find out more about the learners who take part in the Tracker. The first two questions – about age range and gender – cannot be edited. (Nor can the extra question in the Online Tracker, about motivations to study online). The remaining two questions should be edited. They are included to help you group or partition your learners into sub-‐groups for analysis, and generally to tell you more about your sample.

By default we have suggested that you group your learners by stage of study, and by faculty/school (HE) or curriculum area (FE). These are the ways that most of our pilot sites wanted to compare student groups. If you stick with these questions, we still need you to specify what the different options are



called at your institution, or how you will define the different options for the group of learners you are surveying.

You can also decide to change the questions to reflect differences among learners that are important to you. Alternative questions could be:

ñ On which campus are you mainly based?

ñ What is your main mode of study? (e.g. campus-‐based, work-‐based...)

ñ What qualification are you mainly studying for?

Whatever the question, you need to define the answer options carefully to be sure you cover all the alternatives. There is no ‘other’ option with these questions. Note that you cannot delete these two questions: you must decide on a meaningful way to use them.

Some considerations

1. You should choose sub-‐groups that are meaningful in your context and for the population you want to survey. Even if you don’t carry out any comparative analysis, these questions allow you to find out more about your sample. For example, a higher percentage of learners from a particular faculty or age group may respond to the Tracker survey, and that will affect your findings and how you interpret them.

2. An additional consideration arises if you want to compare Tracker data with data from your learning analytics system. You can’t access individual learner responses to the Tracker, but you could associate grouped responses to the Tracker with other, similarly-‐grouped data. For example, you might pool learning analytics data by faculty or subject area. If you want to explore any association between this data and Tracker findings, you should reproduce the same groups in your Tracker data, using one of these questions to do so. (Our Guide to Analysing and Interpreting your Data has more about comparing data sources.)

3. Because you may want to compare different groups of learners – using their responses to these two questions to identify them – it is important that you do not offer too many different answer options. The smaller the groups you divide learners into, the more likely it is that any differences between the groups are the result of chance rather than a real world difference in their digital experience. For a survey of this size, we strongly recommend that you use a maximum of five or six different options*.

4. These options should cover all the groups that learners might fall into.

5. If you use the question about subject of study, for example, your learners may be studying in a large number of departments or curriculum areas. You will have to group these together to



arrive at a manageable set of options. Try to do this in a way that will make sense to learners, and will not leave you with any very small groups.

6. We advise you to change one or both of the grouping questions only if you are confident that:

ñ your new question divides students into groups that are meaningful and useful for understanding their different digital experiences;

ñ you can offer not more than six different answer options, for the reasons already discussed.

* More about group and sample size. Just as with your overall sample, responses from each group need to be representative of all the learners in that group. We know that the smaller the numbers in a population, the higher the response rate needs to be for the responses to be representative. So if you want to make reliable statements about the differences between your groups, you need a valid sample within each group, and you will need a larger overall sample to be sure of achieving this.

Example: If your population is 2000 students, you need 322 responses to consider your respondents statistically representative of the whole group. If you split them into 5 groups – even if those groups are exactly equal sizes – you need 196 responses in each group of 400 for them to be statistically representative of that group, giving a total required response rate of 5 x 196 = 980 rather than just 322.

Because of this we advise you to aim for the highest possible response rate overall, and to divide your sample into no more than four to six groups (that is, offer no more than six options in response to each of these two questions). The fewer the groups, the more likely it is that any differences you find between groups are representative (real) and not just due to chance variations. If it is important for you to be able to make reliable and valid comparisons between particular groups, there is a table to help you calculate required response rates in the Tracker planning sheet.



Your decisions Use this table to note down your chosen text for pages 7 and 8 so that you can refer to it when you are customising your survey in BOS.

Page 7: Optional questions to be included (note any editing required)

Page 7: Prompts for customised question, if used

Page 8: First ‘grouping’ question and answer options

Page 8: Second ‘grouping’ question and answer options

howwillyoucustomise$ the$tracker? - jisc digital...

Documents