Semantic modelsin healthcare education

What is it and how it can improve formative assessments

MedBiquitous Annual Conference 2012

May 2-4 2012 - Baltimore, MD

Muriel FoulonneauYounes Djaghloul

Raynald JadoulNabil Zary

04/19/23 2

04/19/23 3

Challenges

Two main challenges:• Item variability in an assessment

- generate items from a model in order to avoid repeating items- save time and resources, as assessment resource creation is a time and

resource consuming activity

• Learning adaptivity- adapting question forms or assessment path in formative assessment according

to candidate answers or profile

We strive toward: Efficient Approach to Automate/Assist the generation of assessment resources.

19/04/23 OVACS-AIGLE-TAO 4

How the challenges are addressed

19/04/23 OVACS-AIGLE-TAO 5

•Knowledge sources Expert Social / Crowd sourcing Repository Textual

•Knowledge sources Expert Social / Crowd sourcing Repository Textual

Assessment resourcesAssessment resources

•Question generation • Keep the initial semantic• Semantic Inference• Adaptivity

•Question generation • Keep the initial semantic• Semantic Inference• Adaptivity

How ?How ?

In summary the goal was to

• Enable the automatic generation assessment questions based on formal models of knowledge

• Knowledge oriented approach based on semantic technologies:• The creation of a streamline exploring the use of semantic technologies for e-

assessment• Semantic for model checking • Semantic for inference ( to discover knowledge)

19/04/23 OVACS-AIGLE-TAO 6

• Needs to have models with formal representation (such as RDF)• Four questions

- How to build a domain model?- How to validate the proposed model by non IT expert ?- How to generate assessment questions from the refined model?- How to build a flexible delivery environment for these questions?

04/19/23 7

The vision

19/04/23 OVACS-AIGLE-TAO 8

Overview on approach

19/04/23 OVACS-AIGLE-TAO 9

Knowledge: Informal modelsKnowledge: Informal models

Knowledge: Formal modelsKnowledge: Formal models

1.Model building: data mining, human methodology

Formal but not validatedFormal but

not validated

2.Model validation •Experts for model validation •OVACS : to assist experts and to hide the complexity of he formalism (OWL, description logic )

FinalOntology

FinalOntology

Experts, repositories, social media

Experts, repositories, social media

3.Question generation•AIGLE tool, Automatic QTI based questions generation•Semantic similarity techniques

List of assessment questions

List of assessment questions

Validate questionsExperts for question validation

4.Delivery strategy•TAO Delivery Module•TAO QTI viewer

The Final testThe Final test

The process

19/04/23 OVACS-AIGLE-TAO 10

Origins of OAT

19/04/23 OVACS-AIGLE-TAO 11

OVACSOntology VAlidation for Common uSers

How to validate formal knowledge model by questions

19/04/23 OVACS-AIGLE-TAO 12

OVACS: what ?

• Question based strategy for validation• Question to for the validation of the domain not for the assessment• Generate question based on existed knowledge element ( automatic)• More simple for the expert than modifying formal model ( OWL )

• Four ontological components (OC) to validate (RDF schema)• Instance of• All property value• Sub class• Property of a class

• 12 types of feedback• For each OC Accept, remove, don’t Know

• Templates for textual question• Generic (Subject, Predicate, Object)• Dedicated

19/04/23 OVACS-AIGLE-TAO 13

OVACS architecture

19/04/23 OVACS-AIGLE-TAO 14

Source OntologyOWL

Source OntologyOWL

OVACS Engine(Semantic web technologies)

OVACS Engine(Semantic web technologies)

Generated Question(Web based)

Generated Question(Web based)

Ontology of managementOntology of

management

feedbackfeedback

Evaluated ontology Evaluated ontology

Validated ontologyValidated ontology

Expert feedbacksExpert feedbacks

•Manage history•Get past questions•Manage history•Get past questions

OVACS interface

19/04/23 OVACS-AIGLE-TAO 15

http://crpovacscaries.elasticbeanstalk.com/

AIGLEAssessment Item Generator

in Learning Environment

19/04/23 OVACS-AIGLE-TAO 16

AIGLE – Assessment item generator

- Security issue (variability)Adding variability to an item

no expected variation of the construct

- Model-based learning (adaptivity)Generating items from knowledge represented as a model

the construct is modified for each item

19/04/23 OVACS-AIGLE-TAO

17

Stem variables

Options

Key

Auxiliary information

IMS-QTI item generation process

Generating items from Web data sources

19/04/23 OVACS-AIGLE-TAO 18

Calculating the semantic similarity between distractors and the correct answer

Ulan BatorLibrevilleManila

19/04/23 OVACS-AIGLE-TAO 19

Gabon -- Libreville

MaputoPort LouisLibreville

No SemSim With SemSim

Adapted 3 semantic similarity strategies to large scale semantic graphs

Results of user test

Clear decrease of performance in the population when using SemSim (optimizing the similarity between the correct answer and the distractors)

19/04/23 OVACS-AIGLE-TAO 20

User testing with countries and their capital

19/04/23 OVACS-AIGLE-TAO 21

TAOTesting Assisté par Ordinateur

(Computer-Aided Testing)

19/04/23 OVACS-AIGLE-TAO 22

TAO – assessment and feedback loop

The TAO platform is based on semantic web paradigm, i.e. it manages question items decorated with any needed ad-hoc properties

The TAO platform delivers questionnaires that can also be featured with any desiderated extra semantic properties

The TAO collects all answers and behaviors of the test-takers

If extra properties like the “provenance” (i.e. the source model built with OVACS and used by AIGLE) are attached to the question items or to the questionnaire, these properties are stored in tests results

The analysis of the tests results will enforce be used by as feedback loop for a validation process impacting the AIGLE & OVACS phases.

19/04/23 OVACS-AIGLE-TAO 23

OVACSOVACS AIGLEAIGLE TAOTAO

04/19/23 24

Experiment with a dentistry teacher

19/04/23 OVACS-AIGLE-TAO 25

Original hypothesis

The creation of the domain ontology can use semi-automatic strategies, or third party encoders, or a collaborative work: can we ask an expert to validate the assertions in the ontology?

- What is lost in the expert’s speech when creating the ontology?- Does the expert understand automatically generation questions?- Does the expert flag the errors?

19/04/23 OVACS-AIGLE-TAO 26

Creating the ontology

An ontology of the caries- A one hour interview where the teacher explained

the caries, their description, their causes, how to handle them, how to prevent them, how to set a diagnostic

- Definition of a list of concepts / keywords- Creation of classes, instances, and properties- Creation of the OWL ontology

19/04/23 OVACS-AIGLE-TAO 27

Test set upLabels on stand alone

Selected a subset of the ontology to keep the test short: instanceOf (13 items) and subClassOf (11 items)

Only Boolean questions + “I do not know” option

24 questions

2 intentional mistakes: on the content (causes of caries) and spelling (emanel instead of enamel)

Objective: - verify whether the teacher would find the validation mechanism usable- Verify whether errors would be detected and corrected

Video recording of the teacher19/04/23 OVACS-AIGLE-TAO 28

OVACS interface

19/04/23 OVACS-AIGLE-TAO 29

http://crpovacscaries.elasticbeanstalk.com/

Test conclusions

Confusion between

the role of domain expert validating knowledgeand

the role of teacher who prepares questions for students

Objective was not well understood rework experiment conditions

According the comments of our expert:

“Difficulty level of the generated questions is generally low”

“But with very different variations in the difficulty level”

The OVACS validation questionnaire led to:

6 removals (2 subClassOf, 4 instanceOf)

16 accept (9 for subclassOf, 7 for instanceOf)

2 answers “I do not know” for subclassOf meant not relevant

19/04/23 OVACS-AIGLE-TAO 30

Next steps

OVACS

Enrich collaborative features

AIGLE

Ensure a validation / feedback on the generated items

AIGLE generates distractors from an open model (large dataset from the Web) using semantic similarity, but needs to identify relevant distractors in the case of a bounded model (in this case a model for caries)

Predicting item difficulty? Initial test for general culture questions using a Web mining approach. Would need to be tested for medical knowledge.

19/04/23 OVACS-AIGLE-TAO 31

OVACS-AIGLE-TAO

http://tao.lu