Caught in the Web- How young people use the Internet to
resolve civil justice problems
Dr Catrina DenvirUniversity College London
2011 LSRC research looking at use of the Internet for civil justice problems
• Findings were counterintuitive (for some groups, i.e. young people) – High levels of Internet access but low levels of use for problems– Lesser success when used– Even worse for disadvantaged young people
• Fit with existing research: – Resource or ‘toy’ (Nicholas et al., 2003; EdComms 2007) – Willingness correlated with level of educational attainment (Iske, Klein
and Kutscher, 2005) – Difficulty searching/obtaining information related to use of key words
(Landauer 1992)
Research Context
Aims of this Study
What is going wrong?
Finding appropriate information Getting to the right websites? Using appropriate search terms? Assessing trustworthiness of information / jurisdictional relevance? Cross-checking information from a number of sources?
Search behaviour Getting frustrated or giving up easily?
What the Internet was delivering Could the information found replace the need for advice? Was the internet acting as a standalone self-help resource?
Limitations of survey approaches
People may think they remember:
• How long they spent online• The websites they used (although most don’t remember this either)
BUT….not reliable
• Detailed search history / order of search impossible• Not possible to identify errors being made (e.g. inappropriate sources)
Online experiments
An experiment was devised in which young people were presented with legal problems and asked to use the Internet to resolve them
Able to track exactly what they did, the errors they made and the impact of the Internet on knowledge and problem solving behaviour
Structure of Presentation
1. Introduce Experiment
2. How Young People searched online
3. Website behaviour
4. Impact of Internet use on knowledge of rights/strategy
5. What this means for policy
6. Moving Forward – Exploring the content/capability dynamic
Internet Experiment
Two groups targeted
• School aged participants undertaking GCSE’s/A Levels (15-18 years old)• University Students (from a range of disciplines)
208 participants in total
Procedure
Individuals were supplied with a username and password and assigned a specific ‘virtual desktop’ number
Required to read instructions, log into the virtual desktop page, select and run their desktop, and open up the surveyNO time limit
Treat the hypothetical as if it was a problem they were facing themselves
Questions - HousingAlisha rents a house for one year. Discovers the bath has been leaking, causing damp. Landlord is asked to repair. Without providing any notice, the landlord visits the house one afternoon and lets himself in to inspect the leak.
Q1: Is the landlord entitled to enter the house in this way?Q2: Is the landlord obliged to repair the leak?
The landlord refuses to repair the leak. So, after three months ALISHA pays for the repairs and deducts the cost from the next rent payment. ALISHA encloses a note with the rent payment explaining what she has done. After the next rent becomes due, the landlord calls ALISHA and says that she must leave the house in 28 days time. He says she is in breach of the tenancy agreement by not paying the rent in full.
Q3: Has ALISHA breached her tenancy agreement by not paying the rent in full? Q4: If ALISHA refuses to leave, will the landlord be able to evict ALISHA without first obtaining a Court
Order saying that Alisha must leave?
After 28 days, two employees of the landlord arrive at the house to help ALISHA move out.
Q5: Do the two employees have the legal right to enter the property to remove ALISHA’S belongings?
Suppose that before the 28th day, the landlord had obtained a Court Order stating that ALISHA must leave the house by the 28th day
Q6: Would the two employees now have the legal right to enter the property to remove ALISHA’S belongings after 28 days have passed?
Questions - EmploymentALISHA is 19 years old. She has been working 48 hours per week as an employee ZAP COMPUTERS for 1 year. She earns £5.50 per hour. Her manager, PAUL, says he needs her to increase her hours to 50 hours per week. ALISHA does not want to work the extra hours. PAUL shows her a part of her contract which says she can be asked to work up to 50 hours per week.
Q1: Does ALISHA have to work 50 hours per week? Q2: Is ALISHA’S salary, £5.50 per hour, above, below or the same as the National Minimum Wage?
Q3: Does the National Minimum Wage vary according to how old you are?
ALISHA has been asking to see details of the main terms of her contract of employment since she started at ZAP
Q4: Does ALISHA have a legal right to see the main terms of her contract of employment?
One month later – when ALISHA has been working at ZAP for just over a year – PAUL tells her she is going to lose her job
Q5: Is ALISHA covered by the full range of unfair dismissal laws?
Zap’s personnel manager explains that ZAP is reducing the number of technicians it employs, and that ALISHA is going to be made redundant. The personnel manager tells her it is only fair that ‘the younger staff go first’.
Q6: Is ZAP allowed to consider ALISHA’S age in deciding who is to be made redundant?
Internet Experiment - stages
Internet use and access
Socio-demographics
Housing scenario
Questions on rights (1)
Hint website
Questions on rights (2)
Advice preferences
No hint
Questions on rights (2)
Advice preferences
Employment scenario
Questions on rights (1)
Hint website
Questions on rights (2)
Advice preferences
No hint
Questions on rights (2)
Advice preferences
Tracked Activity
OverviewTime Spent
• < 1 minute to < 43 minutes• Mean = 11:46 (SD = 7.56)• Only small differences (<1 minute) between groups on the basis of group
assignment
Searches
• Minimum = 0 : Max: 14• Mean 4 (SD = 2.9)• 19 individuals performed no searches (going directly to the hint website, only
one of these individuals did not receive the hint)• Small differences between searching for those in hint/no-hint groups with the
hint group searching less• Number of searches did not differ between housing/employment groups
Simple• Two to three words, lack of familiarity with the subject-area, or likely to yield
very broad results, e.g. ‘average working week’• Commonly used by those in the housing group
Specific• Familiarity either with the subject area or with the way in which search
engines handled search terms. Typically four terms long, ‘specific’ search terms were distinguished by their sophistication, e.g. ‘national minimum wage UK’, or ‘EU Working Time Directive’
• More often used by those studying law at university
Question• Question format, closed-ended questions aimed at answering a specific
element of the hypothetical problem the protagonist faced, e.g. if my contract says I can work up to 50 hours a week do I have a choice
• More common among those with housing questions, those in no-hint group and school students
Types of Searches
• 54% refined search terms one or more times (on initial failure of original search term)
• No participant went beyond the first page of search results• 85.6% did not use search boxes within websites• Respondents search terms belied a lack of understanding of the nature of the
problem• Law Students were often hindered by their knowledge • Trial and error with little thought given to search terms that might yield the
best results
Search Behaviour
If you are not first, you don’t feature……
• Average of 9 webpages visited• Employment group visited (slightly) more pages than the housing group• Hint group visited a greater number of pages than the no-hint group• University (law) students visited the most number of pages
Websites Used
51.5% of individuals went to commercial
websites
29.5% went to direct.gov.uk
www.gov.uk which 22.5% of
participants visited
which transitioned to
halfway through fieldwork
39% went to AdviceGuide.org.uk
27.5% visited the
Shelter website
30% ‘other government website’ 30% ‘other
third sector website’
16.5% Yahoo Answers/ ‘e-How’/ Discussion Boards
Smaller numbers (<7%) visiting online forums, news sites, blogs and
acas.gov.uk
School students more often used ‘mostly
commercial’ websites
• 35.5 per cent of participants were ‘browsers’• 44.2 per cent of participants were ‘dippers’• 8.6 per cent ‘mixed Users’• School students more commonly engaged in ‘dipping’ behaviour – inability to
specify informational needs?• Where individuals returned to the same website at a latter point in their
browsing, this was often because it had shown up in search results again - brand loyalty?
Website Interaction
Use of the HintEmploymentHint Group - 66.7% of those supplied with the hint used itNo-Hint Group - 44% went on to find the hint website during their searchingHousing Hint Group - 70% of those supplied with the hint used itNo-Hint Group - 40% went on to find the hint website during their searching
• 19.6% visited a website with irrelevant content• 41.5% visited a website with legal content intended for another jurisdiction• Some realised their mistake and added ‘UK’ to the end of search terms• Errors more common among school students
age/inexperience?• School students significantly more likely to arrive at websites with foreign
content. • 53 % of school aged respondents would seek information from one or more
webpages outside of the relevant jurisdiction, v 29.7% of university (law) students and 36.3% of university (other) students
Errors
Impact of the Internet on KnowledgeHousing
• Before - Knowledge poor in regards to eviction without a court order, and whether the landlord’s employees can remove you from the property
• Uncertainty as to what constituted a breach of the lease• After - improvement across the board, but uncertainty about who can evict a
tenant
Employment
• Before - Lack of knowledge in relation to maximum working hours, the minimum wage and who is covered by unfair dismissal
• After - Improvements in knowledge after use of the Internet, yet still unsure as to the rules surrounding unfair dismissal
Improvements, but question dependent
Knowledge of rights and impact of the Internet
Employment
Housing
0 1 2 3 4 5 6
3.3
3.4
4.8
4.6
After InternetBefore Internet
Mean Score Correct out of 6
Impact of the hint
• Hint did not make a difference to score improvement• Housing - hint group achieved lower mean scores after using the Internet than
no-hint group• Hint group viewed more webpages and went to a site where all the
information was contained yet did not improve their scores as much no-hint group.
How can we explain this?• Those with the hint often did not go to other websites• Searching for information in a direct fashion may be beneficial – assuming the
right kind of websites appear in search results• Those using the hint website sometimes used discussion boards – content
issue?
What is the purpose of directing people to a site?
Avoid the pitfalls they would otherwise encounter if left to search alone – assuming that content on the site is delivered in an accessible format
Predictors of improvement
Predicting score improvement on the basis of participant characteristics, search behaviour and website visitation found that:
• School students least likely to improve their scores• University ‘other’ students most likely to improve their scores
• While not statistically significant, those adding UK to the end of search terms were associated with an increase in score, although its contribution was small.
• Other characteristics of searching were not statistically significant (e.g. hint/search type (simple, specific, question), search engine and interaction type (dipper/browser)
• Employment hypothetical, visiting direct.gov.uk was associated with score improvement, gov.uk was not.
• Housing hypothetical, visiting shelter.org.uk was associated with score improvement, other websites were not.
Where should Alisha go (if seeking advice)?
Housing
Broad range of sources of advice
CAB
Lawyer/Solicitor
Free Legal Advice
Union
Local Authority/Council
Friends/Family
Internet
Other
Shelter
ACAS
Don’t know
0 5 10 15 20 25 30 35
After Internet %Before Internet %
% of Respondents
Advi
ce P
rovi
der
Where should Alisha go (if seeking advice)?
Employment
Some changes, though no more coherent
CABLawyer/Solicitor
Free Legal AdviceUnion
Local Authority/CouncilFriends/Family
InternetOther
ShelterACAS
Don’t know
0 5 10 15 20 25 30 35
After Internet %Before Internet %
% of respondents
Advi
ce P
rovi
der
Appropriate action
Before using the Internet
Good knowledge of CAB and the need to get legal advice
But….
Expectation that there are other agencies/ombudsmen/courts that handle problems within an appropriate time frame
Employment• ‘Report it to an authority (home office?)’, • ‘Talk to a government agency that deals with issues of workers' rights’
Landlord• ‘Contact either the police or the local court and explain her reasons for not
moving out or letting the employees in’• ‘Contact a solicitor or ombudsmen, keep any records or proof they tried to
enter’• ‘Get ready to move out and find a landlord who isn't an [expletive]’
Appropriate actionAfter using the Internet
Greater emphasis on informal resolution in respect of employmentRespondents fail to understand legal processes (particularly for housing)Answers do not reflect urgency (interpret with caution)
Employment• ‘Ask her trade union for help’• ‘She should ask a tribunal to decide if she has been discriminated against
because of her age’
Landlord• ‘Get an injunction from the court to stop the eviction notice’• ‘Contact Savilles’ (the estate agents)• ’Leasehold Advisory Service’• ‘Consult the Rent Act’• ‘Get help from your tenants' association…bring up situation at next meeting.’• ‘…..She should contact a property lawyer….’
Increased knowledge of rights not translating to appropriate action
Confidence dealing with problems
Internet may increase knowledge of rights, but does not increase confidenceKnowledge does not equate to confidence/ability to handle alone…
05
101520253035404550
Confidence Handling the Protagonist's Problem Alone
% o
f Res
pond
ents
Problem Handling PreferencesHow would you handle the problem?
Retain preference for family/friends and advisers
Little appetite for handling problems alone
62% would speak to one or both of their parents about the problem88.5% said they would use the Internet to obtain information about rights
57.2
20.7
13.9
5.8
Family/friends AND an advisor/representativeAdviser/representativeFamily/friendsOn my ownNothingI'm unsure
Summary
Internet increased knowledge of rights• Question dependent• Many still got answers wrong, despite availability of resources
Searches were structures around search engines• Not as much searching within sites as might be expected • Errors stemming from search terms• Some lack of awareness regarding jurisdiction• Little attempt to discriminate between sources of information• Rigid adherence to order of search results
In addition...
Knowledge does not equate to appropriate action• Little change in knowledge of relevant processes / appropriate action• Internet did not increase understanding of advice sources
Knowledge does not equate to confidence• Continue to rely on family / friends / advisers, valuing their experience• Seemingly little appetite for self-help
Implications
For public legal education and self-help (in light of policy changes)• Internet blunt instrument for PLE in the context of this experiment• Knowledge does not equal confidence or competence with regard to action
For design of websites• The best designed websites will receive little attention if they are not first in
search results (and young people did not take direction well)• Future developments and website functionality• Phrasing of search terms has implications for website designers
Policy findings • Problems likely to be exacerbated for vulnerable young people• 1st and 2nd digital divide• Less reliable support structures
Internet appears to be being used as a diagnostic tool to confirm the nature of a problem before advice is sought from another source (parent/advisor).
We do not know if this is to do with the content available online or the way it is perceived by audiences
Capability or Content – Search Experiment
• Final part of study looked at whether varying search terms (in relation to the hypothetical questions) led to different websites and websites of different quality.
• The results emerging from the internet experiment offer less comfort than might be hoped in respect of the capacity of a technologically proficient group of young people to devise search terminology.
• A lack of familiarity with the context of the hypothetical problem rendered participants predisposed to relying on cues drawn from the rights-based questions they were asked.
• Where such cues are not available (as may well be the case in everyday problem solving) young people may struggle to conceive of search keywords.
• We know that ‘simple’, ‘specific’ and ‘question’ based search terms did not have a (statistically) significant influence on whether or not participants’ improved their knowledge following Internet use.
• But, do the different search terms lead to search results of a different nature or quality?
• In the search experiment study, 580 web pages examined – some of which were duplicates
• ‘Simple’, ‘Specific’ and ‘Question’ based search terms were devised for each one of the hypothetical questions
• Top ten Google and Yahoo search results for each search term/phrase were assessed against an evaluation framework
• Framework based on similar framework used by Advicenow in 2006 and more general indicators of quality/credibility drawn from the literature
Search Experiment
Additionally Classified According to whether:Government/Third Sector/Commercial/Union/News Site/Discussion Board/Blog/Other
Mean problem support, skills support and credibility scores were typically higher for government and third sector websites than for other website types.
Credibility and problem support scores were far higher for union websites than for any other type of website in relation to the employment hypothetical.
In respect of the housing hypothetical, commercial websites tended to score far lower on problem support than commercial websites evaluated for the purposes of the employment hypothetical.
Overall, irrespective of the type of website, all fared quite poorly in terms of skills support. Only a small number of websites (20.1%) met the criteria for ‘providing information specific to young people’, most often, these websites were provided by the government (58.1%).
Search Results- Quality
Compared to use of a ‘simple’ search term, a ‘specific’ search related to a significant increase in the likelihood of a relevant result.
There was also an increase in the likelihood of a relevant result for question based searches (when compared to ‘simple’ searches) though the difference fell well short of statistical significance.
Compared to Google, Yahoo was significantly less likely to yield relevant results. A Google result could be expected to be relevant around 70 per cent of the time compared to 59 per cent for Yahoo.
Google/Yahoo relevance is likely tied to the fact that the presence of commercial websites in the Yahoo result listings was in some cases more than double that of the Google listings for both the employment and housing searches.
Question based search terms more commonly led to discussion board, ‘communities of knowledge’ type information.
Search Results- Relevance
• Search Engines are absorbing some of the ‘bad habits’ of young people when it comes to searching. Ten years ago a question based search would have been rejected, now search algorithms are being designed around these behaviours
• Although search behaviour does not have as much of an influence as expected in this field, this might be due to the relatively small number of websites operating in the legal advice space.
• Nevertheless, search behaviour can lead to less desirable websites and if individuals lack the capacity to distinguish between what is and what is not a reputable site, then this is not ideal.
• If it is expected that more young people will be using the Internet to self-help, then PLE and technological capability go hand in hand
• The idea that technology will compensate for any lack of capacity on the part of the user is appealing in its simplicity, but this simplicity ultimately underestimates the centrality of the user to the process of civil justice self-help.
What can we conclude?