education and skills online
TRANSCRIPT
0
Education & Skills Online
What is Education & Skills Online?
• The OECD Education & Skills Online offers individuals, businesses, community based and government organizations a powerful new set of tools to assess their adult populations.
• It is specifically designed to provide individual level data that is linked to PIAAC and, therefore, shares the same validity evidence
• It measures both cognitive and non-cognitive skills
• It uses an Internet delivered platform that is available 24 / 7 and includes technical support
• It was developed and field tested in 9 language versions in 2013, and there will be a subsequent field test for additional countries who may want to join
1
Education & Skills Online Contents
• Core background questionnaire
• Cognitive modules:
• Literacy and numeracy assessments with linking PIAAC items and new items developed specifically for Education & Skills Online• 53% linking items and 47% new items
• Reading Components module to evaluate the literacy skills of low-performing readers• Evaluates print vocabulary, sentence processing, and passage
comprehension
• Problem Solving in Technology Rich Environments module
2
Education & Skills Online Contents Continued
Noncognitive modules:• Career Interest and Intentionality - This module measures
an individual’s preferences for different types of work activities and environments and the level of an individual’s intention to seek out new job opportunities and career-and job-related training.
• Skill Use – This module utilizes items from PIAAC to assess the specific skills that respondents use in both their work and daily lives as important drivers of skill acquisition as well as critical outcomes affecting their lives. The questions in this module focused on skills associated with reading, writing, use of mathematical information and ideas, and information and communications technology (ICT).
• Subjective Well-Being and Health - This module examines the main components of subjective well-being: life evaluation and positive and negative affect, in addition to subjective health and well-being indicators.
• Behavioral Competencies (Forthcoming)- This module measures selected personality facets based on their high relevance and utility for academic and workforce readiness and success.
3
Uses and Benefits
• Individual scores in each measured domain provided automatically and in real time
• Incorporates item response theory, multi-stage adaptive testing and automated scoring
• Individuals and organizations have flexibility to determine which skills to assess
• Organizations purchasing tests will have access to their database of results for further analysis
4
Package Types
Customers have the option to purchase codes in three package types, described here.
5
Package Contents
Cognitive • Background questionnaire• Literacy and Numeracy assessments with adaptive testlets• Reading components• Problem solving in technology-rich environments
Noncognitive • Background questionnaire• Career Interest and Intentionality• Skill Use• Subjective Well-Being and Health
Bundled Cognitive and Noncognitive
• The contents of both the cognitive and noncognitive packages with results linked by the authorization code
Workflow of Education & Skills Online
6
How Test Takers Access Assessment
Each test taker will receive a unique 10-digit test authorization code from the customer who purchased the codes.
The customer will direct the test taker to the Education & Skills Online log-in page (next slide) where the test taker will enter the test code twice to log in.
In the upper right hand corner of the log-in page, test takers or administrators can use the drop-down menu to change the language of the log-in page. The language of the test is determined by the test authorization code.
Test takers MUST use the Firefox browser to take the assessment.
7
8
Change language
Home Page
9
Background Questionnaire
1. Age
2. Gender
3. Highest Education Level
4. Were you born in [Country]?
If no, in what year did you come to live in [Country]?
5. What is your first language?
Language typed in if answer is “other”
6. Current employment status
Current occupation (ISCO-08 code)
10
Starting the Core Cognitive Assessment
11
Core Assessment Workflow Detail
12
Literacy and Numeracy Testlet Formation
• Each test taker will be presented with two testlets for literacy and two testlets for numeracy
• A testlet is a set of units grouped to target the low, middle, and high end of the PIAAC literacy or numeracy scale
• Testlets overlap so that a few items are shared by two testlets
13
Testlet 1
Testlet 2
Testlet 3
◄ Easier - - - - - - - - - - - - - - - Harder ►
Literacy and Numeracy Assessment
• The literacy and numeracy assessment is expected to take 60 minutes.
• The test uses the test taker’s response to certain background questions to determine which stage 1 testlet to present.
• There are 9 questions in stage 1 for both literacy and numeracy.
• The test taker’s score on the first testlet is used to choose the stage 2 testlet.
• There is no transition screen between the first and second testlet– it appears seamless to the test taker.
• There are between 11 and 13 questions in the stage 2 testlets.
14
Resuming an Assessment
15
Score Reports
• The customer decides whether the test taker will see a score report at the end of the test or whether the score report will be saved in the online system that can be accessed by test administrators.
16
Score Reports for Literacy and Numeracy
• Include an explanation of the skills that were measured based on PIAAC proficiency level standards and how those skills are used in everyday life
• Characterize the strengths and weaknesses in the skill areas assessed
• Provide a scale score for the individual test taker
• Provide a comparison of the test taker’s score to the national and international data from PIAAC by education level, occupation, and age group
17
18
Literacy and Numeracy Score Report
Optional Cognitive Assessments
19
Optional Cognitive Modules Workflow Detail
20
Reading Components
• Includes three sections: • Vocabulary (38 items)
• Sentence Comprehension (22 items)
• Passage Comprehension (44 items)
• Test takers are given two scores for each section: one for their rate and one for their accuracy
21
22
Reading
Components
Score Report
Problem Solving in Technology-Rich Environments
• Begins with a short orientation for the test taker
• 9 questions from PIAAC
• Test takers see a score report after completing the test
23
24
Problem Solving
in Technology-
Rich
Environments
Score Report
All Assessments Completed
25
• When the test taker completes all the modules in the workflow, all the modules will appear in the Completed Assessment table.
• Clicking on the name of the completed module will open the score report for that particular module (if the score reports are turned on for the institution).
Noncognitive Module Workflow Detail
26
Skill Use+
Score Report
Career Interest and
Intentionality+
Score Report
Subjective Well-Being and
Health+
Score Report
If noncognitive modules are included in the purchased product package, then the test taker has the option to
take the following modules in any order:
Noncognitive Modules
27
Skill Use
• 57 items (from the PIAAC background questionnaire)
• Takes approximately 5 minutes to complete.
• Assesses the specific skills that respondents use in both their work and daily lives as important drivers of skill acquisition as well as critical outcomes affecting their lives.
• The questions in this module focused on skills associated with reading, writing, use of mathematical information and ideas, and information and communications technology (ICT).
28
Career Interest and Intentionality (CII)
• 92 questions
• Takes approximately 10 minutes to complete.
• This module measures an individual’s preferences for different types of work activities and environments and the level of an individual’s intention to seek out new job opportunities and career- and job-related training.
29
Subjective Well-Being and Health
• 27 questions
• Takes approximately 5 minutes to complete
• Research has shown the predictiveness of health for education and work-related outcomes, with healthy individuals being more productive and making less use of health care.
• This module examines the main components of subjective well-being: life evaluation and positive and negative affect in addition to subjective health and health indicators in line with the measures described in the World Health Organization’s agenda.
30
Pricing and Test Administration Portal
31
Pricing for Package Types
• Package 1: Cognitive Modules
• Package 2: Noncognitive Modules
• Package 3: Bundled Cognitive and Noncognitive
32
QuantityPurchased
Unit Price Package 1
Unit Price Package 2
Unit Price Package 3
<5,000 €9.00 €2.00 €11.00
5,000 – 10,000 €8.25 €2.00 €10.25
10,001 – 25,000 €7.75 €2.00 €9.75
25,001 – 75,000 €7.00 €2.00 €9.00
75,001 – 149,999 €6.00 €2.00 €8.00
150,000+ €5.00 €2.00 €7.00
Test Administration Portal
• Purchasers will have access to an online portal to:
• Monitor test completion
• Download result data in Excel spreadsheets for analysis • Background questions, scale scores in literacy,
numeracy, and problem solving, and noncognitivemeasures
• Create additional users for their organization
• Create up to two organizational levels below the purchasing institution
• Read technical documentation
33
Functionality in Test Administration Portal
• Create new agencies and locations and assign codes to those groups
• Download Excel spreadsheets with lists of activated codes
• View available/used code inventory for each agency or locations
• Reclaim unused inventory from an agency or location that can be reallocated to another agency or location
• View and print score reports
• Create additional users at each level:• Administrators – access to all functionality, including downloading data and
creating new users
• Operations Support – ability to download authorization code spreadsheets and assign test codes to sub-levels, but not to download data or create new users
34
Distribution of Test Authorization Codes and Data Access
• Purchasers can choose to create sub-levels to manage test authorization codes
• Purchasers can distribute codes either by allowing these levels to draw down codes automatically or requiring them to request codes through the level above them
• Users at each level can be either administrators with full access to all features or operations support with the ability to distribute codes but not download data
• Each level can download data for its institution and the levels below its institution
35
Location
Agency
Organization (Purchaser)
Organization (1,000 codes)
Agency
(500 codes)
Location
(250 codes)
Location
(250 codes)
Agency
(500 codes)
Location
(250 codes)
Location
(250 codes)
Data Download
• Only users with administrator rights are able to download data in an Excel file format.
• Data can be downloaded on demand. Users do not need to request data from ETS.
• When downloading data, users can use search features to specify a date range or use pre-designated time rages such as “all data from the last week.”
• The authorization code is the unique identifier of the individual test-taker. No personal information (names, email addresses, phone numbers) is collected by Education & Skills Online; therefore, no personal information is stored in the system.
36
Data Sharing
• The purchaser will own the data and therefore has the responsibility of notifying individuals of their rights and obtaining any permission necessary for sharing the data that has been collected.
• Any entity (country, institution, etc.) can become a purchaser in order to maintain control over data and privacy concerns.
• The purchaser may send data to other entities, with proper permission from the individual test takers, but must do so outside of the test delivery system.
• A purchaser may arrange to share data with the OECD, but any agreements would be the responsibility of the purchaser in cooperation directly with the OECD. It will be the purchaser’s responsibility to supply the data to the OECD in the agreed upon format and conditions.
37
Data Download Fields
General Information Data Fields
(for all package types)
• Authorization Code
• Institution Name & ID
• Language
• Background questionnaire answers (9)
• Date and time the code was first used
• Start date and time (for each module)
• Last accessed date and time (for each module)
Core Cognitive Data Fields
• Literacy and Numeracy scale scores
• Literacy and Numeracy level
• Reading Components scores (rate & accuracy) for vocabulary, sentence comprehension, and passage comprehension
• Reading Components levels for rate and accuracy for the three components
• Problem Solving scale score
• Problem Solving level
38
Noncognitive Data Download Fields
Subjective Well-Being and Health
• Life Satisfaction, Positive Affect, and Negative Affect levels (low/medium/high)
• Subjective Health level (good/fair/poor)
• BMI class
• Sleep, Diet, Smoking, and Physical Activity levels
Skill Use
• Levels for Reading, Writing, Numeracy, and ICT at work and at home (8 scores)
• Responses to 6 questions about computer use
Career Interest & Intentionality
• Scores for each of the 6 interest categories
• ISCO codes and job fit level for current and intended jobs
• ISCO codes for the 20 best-fitting jobs
• ISCO codes for the 10 poorest-fitting jobs
• Levels for questions about intention of finding a new job/seeking training
39
Find Out More About Education & Skills Online at:
www.oecd.org/skills/ESonline-assessment/
Follow us on:www.facebook.com/OECDSkillsSurveys
Education & Skills Online is a joint initiative of the OECD, participating countries and the European Commission. The development of the assessment
was possible thanks to the financial support of the European Commission. The delivery of and user support for the assessment is provided by Education
Testing Services (ETS) on behalf of the OECD
