Development of Evaluation Instruments based on ICT Skills Standards for Teachers (ISST)

7th May, 2014

CONTENTS

I Request of MOE

II Proposal for Evaluation

III Development Strategy & Schedule

IV Results & Expectation

I Request of MOE

Request of MOE

Mixed Model(method)

CategoryEvaluation

Set 1 Set 2 Set 3

Category A Likert Scale

Category B Multiple (a) Multiple(b) Multiple(c)

Category C Multiple (a) Multiple(b) Multiple(c)

Category D Likert Scale

… … … …

Definition of Mixed Methods Evaluation

Qualitative Data

Mixed Method

Quantitative Data

• Instruments• Checklist• Records

• Interviews• Observations• Documents• Audio-visual materials

Mixed methodology is a design for collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series for studies to understand an evaluation problem(Creswell and Plano Clark, 2007).

II Proposal for Evaluation

Proposal for Objective and Valid Evaluation

Multiple ChoiceEstimate scientific, directive, and objective performance.Take long time to develop the test sheet and also difficult

to develop identical test sheets. Respondents’ position, difficult to respond.

Before developing the test times, it is necessary to check the characteristics of respondents

Likert ScaleEasy to verify the reliability and validity of test items.

Respondents’ position, Easy to respond. The same test items can be used.

It’s difficult to estimate real ability because it is based on the respondent’s perception.

Respondents can misrepresent about their ability

EvaluationInstruments

Likert Scale

Multiple Choice

• Master Teacher• Leader Teacher

• Leader Teacher

Evaluation Framework of ISST

CompetenciesCompetency Factors

Accessibility Capability Practical Use Satisfaction

Technology

Communicative

Pedagogical

Research /Investigative

Management

Design

Domain of ICT competence for teachers is technology, communicative, pedagogical, research/investigative, management and design.

Teacher Competencies for ICT Use in Education

According to the MEN (2013), there are three levels of all teachers’ competencies: Explorer, Integrator, and Innovator.

CompetenciesLevels

Explorer Integrator Innovator

Technology

Communicative

Pedagogical

Research /Investigative

Management

Design

III Development Strategy & Schedule

Development Strategies and Schedule

Develop & Confirm Indicators• 5.10 ~ 5. 20

Expert review• 5.20 ~ 5. 30

Preliminary test with 300 samples • 6. 1 ~ 6. 7

Analysis and set up the tests • 6. 8 ~ 6. 15

Development oftaxonomy• 6. 1 ~ 6. 10

Item development• 6. 10 ~ 6. 25

1st Preliminary test With 300 samples • 6. 25 ~ 7. 5

Analysis and revise the tests • 7. 5 ~ 7. 10

Setting the criteria6. 15 ~ 6. 25

2st Preliminary test With 300 samples • 7. 10 ~ 7. 20

Analysis and set up the tests • 7. 20 ~ 7. 25

Setting the criteria7. 25 ~ 7. 31

Multiple Choice

Likert Scale

Development Methodology

Validation of Evaluation Instruments

Content validity (through experts’ review) Construct validity (through statistic program) Item response theory (through statistic program)

Method for Setting the Criteria ( Standard-setting methods)

Angoff & Jaeger method

IV Results & Expectation

Representation of Evaluation Result

Radial Chart

Technology

Communicative

Pedagogical

Research / Investigative

Management

Design

0

2

4

Average

Investigative

Setting the Level of Teachers based on Criteria

InnovatorIntegratorExplorer

Score

Score

Score

Output

Likert Scale At least 72 items

Multiple Choice At least 100 items

ISST & Evaluation Items Development

ISST & Evaluation Items Development

Draft ISST Test Item

( 4. 23 ~ 5. 8 )

Development of ISST

( 3.4~4.30 )Conduct Expert

Review( 5. 8 ~ 5. 15 )

Validate Test Items( 5. 15 ~ 5. 30 )

Construct Test Sets

( 5. 30 ~ 6. 5 )

Review and Confirm Test Sets

( 6. 5 ~ 6. 11 )

Pilot Test and Port the Online Test

System ( 6. 11 ~ 6. 17 )

Submit Test sets

( 6. 17 )

Prepare Draft Report

( 6. 17 ~ 7. 25 )

Review by MOE and Revise the

Report ( 7. 25 ~ 8. 18 )

Submit Final ISST Research

Report( 8. 18 )

ISST & Evaluation Items DevelopmentConstruct Test

Sets( 4. 30 ~ 5. 5 )

Development of ISST

( 3.4~4.30 )Review and

Confirm Test Sets( 5. 5 ~ 5. 10 )

Draft Likert Scale Item

( 5. 10 ~ 5. 20 )

Conduct Expert Review

( 5. 20 ~ 5. 30 )

Preliminary Test with 300 samples

( 6. 1 ~ 6. 7 )

Analysis and Set Up the Test

( 6. 8 ~ 6. 15 ) Submit Likert Scale Test set

( 6. 17 )

Prepare Report & Revise

( 6. 17 ~ 8. 18 )

Submit Final ISST Research

Report( 8. 18 )

Setting the Criteria ( 6. 15 ~ 6. 25 )

Development of Taxonomy

( 6. 1 ~ 6. 10 )

Item Development ( 6. 10 ~ 6. 25 )

1st Preliminary Test ( 6. 25 ~ 7. 5 )

Analysis and Revise the Test ( 7. 5 ~ 7. 10 )

2st Preliminary Test ( 7. 10 ~ 7. 20 )

Analysis and Set Up the Test

( 7. 20 ~ 7. 25 )

Setting the Criteria ( 7. 25 ~ 7. 31 )

Submit Multiple Choice Test set

( 7. 25 )

Question & Answers