how students judge the relevance and reliability of information in the digital environment

How students judge the relevance and reliability of information

in the digital environment

Curtis Watson

ALIA National Library & Information Technicians Symposium, 1 November 2013

From: http://2.bp.blogspot.com/_nRfMpxrkFJU/S90JUgWQksI/AAAAAAAAAVk/Xwk3isGSMp8/s1600/brain%2520puzzle.gif

Participants

Secondary school students undertaking information search tasks

• 37 students• Between 14 and 17 years of age • Years 9 to 11• From a school in south-eastern Australia

From: http://us.123rf.com/400wm/400/400/stockbroker/stockbroker0806/stockbroker080604128/3204223-secondary-school-students-in-a-school-hallway.jpg

Data

• Students’ journals• Structured and semi-structured interviews• Think-aloud reports and video screen captures• Video-stimulated recall interviews• Questionnaires

From: http://www.bgsu.edu/departments/greal/llc/germanwq/Germ670_AYASalzburg/images/Interview.jpg

From: http://www.freewareguy.com/wp-content/uploads/2009/11/screencapture.jpg

From: http://languageartsgames.4you4free.com/word_analysis.gif

From: http://bcdigital.com/media/portfolio_finals/bullseye.jpg

‘the top one [is] easier to click’

‘with Google, the first one normally is the one you’re looking for’

(Sophie)

‘it was the first one on the top of the list’

(Victoria)

Sometimes the words are […] like they just have like a ‘Rome’ over here and a ‘dependence’ over there and […] and I just, I don’t go to them, I go to the ones where the words are like what I want and they’re all close together kind of thing.

(Karen)

‘if the title, like the blue thing, matches what I’ve typed in, then I just go to it’

(Karen)

‘Quite certain that a composer like him would have his own page ‘

(Paul)

‘if it was very cluttered and a bit over my head, then I sort of went away from that to a more, to information that I could understand ‘

(Richard)

‘it’s either going to mean I’ve got to buy it or I’m not really going to be able to view much of it’

(Sharon)

‘Ah, Amazon, I don’t want to have to, I can’t buy a book’

(Chloe)

‘Generally I just tend to skip videos’

‘it’s just harder to analyse [videos] and, if I’m doing an assignment, I just want what’s basic, given to me in plain English’

(Stefano)

‘people’s opinions on analysis stuff generally aren’t very good’

(Gerard)

‘usually I just go for the basics first, so I’ll get a basic outline of what I’m trying to find out and then I’ll just build on it after a few researching sessions’

(Mary-Ann)

Chloe‘I generally start with Wiki, just if, I don’t know, just to give me a general idea’ … ‘Wikipedia, good place to start’

David‘Wikipedia gave me good background information about both the topics’

Edward‘It’s [Wikipedia’s] just a starting reference point for me’

Emily‘I always use it [Wikipedia] to get a summary of the things’

Jenny‘Wikipedia […] I just go to for a general overview’

Lionel‘I would tend to use a Wikipedia article, not as the only source but as something you can refer to and you can get a grounding knowledge in’

Mark‘I just get a base knowledge from Wikipedia […] it just starts me off, ’cause it always has like the start of everything’

Paul ‘Let’s have a look at Wikipedia to get just a general idea’‘Wikipedia’s useful in terms of, um, concepts or topics which you have absolutely no idea, you just need a brief introduction to it’

Once you have a good site, sometimes you’ll take a piece of information from it, and search it from that, so like the physicist, so maybe I’ll search him from that first site that gave me the information on him. […] What I’d next do is just have a read through, see what else I can find and then search from that new information.

(Mary-Ann)

‘they just went through it more thoroughly’

(Elizabeth)

‘they had the most information in them’

(Sophie)

sort of scan-reading like the first sentence of each sort of paragraph and if it was, appeared that it would be helpful, I continued reading the paragraph and stuff’

(Victoria)

From: http://www2.tfk-racoms.com/racoms/_picture/content/reliability.jpg

‘There was just so much and it was a ‘.gov’ site […] so that usually does mean that it is reliable’

(Kristian)

‘weird-looking website but we’ll go to it, it’s got a dot org, I like dot orgs’

(Chloe)

‘BBC, very, very reliable’

(Stefano)

[Referring to Geoscience Australia’s website] ‘really, really reliable’

(Gerard)

‘It’s by the […] Geological Society, so they have a lot of, like, people contributing, so it’s not very biased but it’s got a lot of facts in it, which is good for this, um, I trust it, I do, and they have a very long reference list, which is also nice when you’re looking for things; looking through their references is also very good, so I think that’s a very trustable source.’

(Gerard)

‘I’ll normally just go to the, take the information from that, double-check it with, um, newspapers and independent articles that have been written’

(Edward)

‘I compared this information to the basic information I had got from the Wikipedia site’

(David)

‘they don’t look like there’s much design put into them … so I look at them and I think, well, that’s probably not the best one to choose’

(Richard)

‘I think if it’s too showy and has like too many advertisements, like, I’d kind of stay away, wouldn’t stay away, […] but I’d be, like, “This probably isn’t as, like, dependable as, like, a boring one” ’

(Karen)

‘[a] bit gimmicky but not too bad’

(Edward)

‘if it was poorly written, I probably wouldn’t bother with them, even, no matter how good the, um, author is’

(Edward)

‘it sounded like a really good argument … he actually had evidence ’n’ discussed it in depth and stuff’

(Karen)

‘I don’t usually check for the authors’

(Karen)

‘where the information comes from, it’ll usually say on the web page, if it’s come from a specific person, um, it might be a person who’s well-known or a person that’s, um, you know, properly trained, like a professor or something particular’

(Charlotte)

‘if it says like a doctor wrote it or something, […] it needs to actually have proof that it’s a really good, you know, piece of work’

(Michael)

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks• research process unchallenging

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks• research process unchallenging• least challenging path to task completion

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks• research process unchallenging• least challenging path to task completion• depend on relevance rankings of search engines, favouring results that occur

early in a set, and do not demonstrate any understanding of the methods used by the search engine

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks• research process unchallenging• least challenging path to task completion• depend on relevance rankings of search engines, favouring results that occur

early in a set, and do not demonstrate any understanding of the methods used by the search engine

• heuristics save the effort of accessing the full resource

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks• research process unchallenging• least challenging path to task completion• depend on relevance rankings of search engines, favouring results that occur

early in a set, and do not demonstrate any understanding of the methods used by the search engine

• heuristics save the effort of accessing the full resource• intuitive judgements

Theoretical statement 1

Convenience and pragmatism

• convenient and pragmatic approach to information search tasks• research process unchallenging• least challenging path to task completion• depend on relevance rankings of search engines, favouring results that occur early in a

set, and do not demonstrate any understanding of the methods used by the search engine

• heuristics save the effort of accessing the full resource• intuitive judgements• prevalence of information for relevance judgements and deciding topic’s main points

Theoretical statement 2

Search for an overview

• strongly motivated to find overview of topic of interest

Theoretical statement 2

Search for an overview

• strongly motivated to find overview of topic of interest• provides framework for learning from information sources found later

Theoretical statement 2

Search for an overview

• strongly motivated to find overview of topic of interest• provides framework for learning from information sources found later • early adoption of information at hand may prevent deeper/broader exploration of

topic

Theoretical statement 3

Incidental nature of establishing reliability

• information accords with that found in earlier sources

Theoretical statement 3

Incidental nature of establishing reliability

• information accords with that found in earlier sources• students’ perceptions that teachers consider Wikipedia unreliable

Theoretical statement 3

Incidental nature of establishing reliability

• information accords with that found in earlier sources• students’ perceptions that teachers consider Wikipedia unreliable• perceptions do not prevent use because provides overview of topics

Theoretical statement 4

Prior knowledge

• two forms: information remembered from an earlier experience preconceptions of ability of genre, or information provider, to offer relevant

and reliable information

Theoretical statement 4

Prior knowledge

• two forms: information remembered from an earlier experience preconceptions of ability of genre, or information provider, to offer relevant

and reliable information • reputation of a source’s reliability based on a generalised personal belief (outcome

of socialisation or personal experience?)

Theoretical statement 5

Process of building knowledge

• filtering of information occurs in process of relevance judgements

Theoretical statement 5

Process of building knowledge

• filtering of information occurs in process of relevance judgements• rare explicit judgements of reliability of information; depend on impressions of

trustworthiness

Theoretical statement 5

Process of building knowledge

• filtering of information occurs in process of relevance judgements• rare explicit judgements of reliability of information; depend on impressions of

trustworthiness • matching employs judgements of relevance and reliability simultaneously

Theoretical statement 5

Process of building knowledge

• filtering of information occurs in process of relevance judgements• rare explicit judgements of reliability of information; depend on impressions of

trustworthiness • matching employs judgements of relevance and reliability simultaneously• adding to finished product mostly occurs with little apparent discrimination

Implications for practice

• checklist of features to gauge reliability of information not consonant with naturalistic decision making

• information adopted because of its early appearance ought to be questioned in later stages of the ISP

• prior knowledge should be harnessed more explicitly in the learning process

• background knowledge should be developed from more than one source.

From: http://mhpbooks.com/mobylives/wp-content/uploads/2010/07/ebooks1.jpg

Follow-up study?

I use Wikipedia to give me an introduction to my topic.

Wikipedia is a reliable source.

When I find information in one source, I check that information with information in another source.

I trust websites that have .org in their URLs.

Follow-up study?