the pépite project
DESCRIPTION
The Pépite project. Evaluating the performance of a diagnostic system in School Algebra. Naima El- Kechaï , Élisabeth Delozanne , D. Prévit, B. Grugeon, F. Chenevotot. Outline. Why a cognitive diagnosis ? The Pépite project The diagnostic system A simple example PépiDiag Evaluation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/1.jpg)
1
The Pépite project
Naima El-Kechaï, Élisabeth Delozanne, D. Prévit, B. Grugeon, F. Chenevotot
Evaluating the performance of a diagnostic system
in School Algebra
![Page 2: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/2.jpg)
Outline Why a cognitive diagnosis ?
The Pépite project The diagnostic system
A simple example PépiDiag
Evaluation Conclusion
2
![Page 3: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/3.jpg)
The Pépite project Objective
To design and implement a web-based application To support math teachers To manage the students’ cognitive diversity
Domain Algebra class 9th-10th grade (15-16 years old)
3
![Page 4: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/4.jpg)
A teaching scenario Who ?
Mary is a Math teacher in 9th grade Context
To bring all her class to the same level Steps
1. Students have the online diagnostic test2. Pépite
a) analyzes the students’ answers b) displays a cognitive profile of each student and of
the whole classc) suggests 6 groups of students and a session of
exercises adapted to their cognitive profiles3. Mary can adjust the groups and the exercises4. Students log in and do the exercises in their assigned
session 4
![Page 5: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/5.jpg)
A cognitive profile of a classGroupe A+
Engage in algebraic thinking
Use algebraic calculation appropriately
Groupe B+ Begin to use algebra to
solve problems Use sometimes mal-rules
Groupe C- Stuck in arithmetical
thinking Algebra makes no sense
5
![Page 6: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/6.jpg)
A B+ student’s cognitive profile
6
Level Personal featuresAlgebraic calculation Techniques
Sucess rate on asked questions: 4/12Interpreting algebraic expressionsSuccess rate on asked questions: 11/23
Strong pointsPartial mastery of algebraic rulesSome good interpretations of algebraic expressions
Weak pointsLow mastery of algebraic calculation
Usage of algebra Mathematical modelingSuccess rate on asked questions: 5/9
Strong pointsGood mastery of algebra usage on some problems
Weak pointsJustification by school authority
Algebraic translation Translating situations to algebraSuccess rate on asked questions: 12/24
Strong pointsSome good translations of mathematical relations
Weak pointsSome translations without reformulation
More
More
More
More
More
More
![Page 7: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/7.jpg)
A differentiated session
7
![Page 8: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/8.jpg)
Pépite : Cognitive Diagnosis Process
8
![Page 9: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/9.jpg)
A simple example
9
![Page 10: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/10.jpg)
Answer assessment (local diagnosis)
10
Assessment Dimensions Values CodesValidity Incorrect V3Use of Letters Incorrect L3Translation Incorrect T332Algebraic Expressions writing
turning a product into a sum a*a-> 2a
EA41
Justification By algebra using incorrect rules J3Type of answers :
Recognition of sub-figures but turning a product into a sum
![Page 11: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/11.jpg)
Exercise coding prescription file
Student’s answers Answers coding+ Student’s profile
PépiDiag general architecture
11
Student’s interface
PépiDiag
Teacher’s interfaceStudent
Database
Student’s answers
Exercise Database
Solves exercises
![Page 12: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/12.jpg)
A coding prescription file (partial)Type Label Code Patterns of
equivalent expressions
1 Correct expression of the area as a product Length × Width
V1, T1,J1 (a + 3)×(b + a)
2 Correct expression of the area by adding the areas of the different rectangles
V2, T1, J1 ab+a²+3b+3a; (a×(a+b))+(3×(a+b));a(a+3)+b(a+3)
3.1 Recognition of sub-figures with parentheses errors (e.g. a+3 a+b)
V3, EA3, T3, J3 (a+3)×b+a; a+3×(b+a); a+3×b+a; a+3×a+b; a+b×a+3
3.2 Recognition of sub-figures with gathering transformations
V3, EA42, T3, J3 (a+3)(ab); (b+a)×(3a); a+3 × ab ; ab×a+3; 3a×a+b ; a+b×3a
3.3 Recognition of sub-figures but turning a product into a sum a²->2a
V3, EA41, T3, J3 5a+ab+3b; ab+2a+3b+3a
…9 No interpretation Vx 12
![Page 13: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/13.jpg)
PépiDiag
• Local Diagnosis
• Exercise coding
prescription file
• Coding of student’s answers
• Computer• Algebra
System
• Student’ s
answers file
13
• Global Diagnosis
• Student’s profile
![Page 14: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/14.jpg)
Outline Why a cognitive diagnosis ?
The Pépite project The diagnostic system
A simple examplePépiDiag
Evaluation Conclusion
14
![Page 15: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/15.jpg)
Evaluation Quality requirements
1. No correct answer badly coded2. Better no code than a wrong one3. Minimal number of answers left unanalyzed
Method Comparison between experts & system
assessment 3 experts assessed 360 answers of each exercise
(one by one) Experts worked separately and then agreed on
some corrections to minor errors
15
![Page 16: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/16.jpg)
Unanalyzed answers
16
Empty answers
Students’ answers
94/360 266/360
Data collected (on the exercise used as an example)
PépiDiag analyzes • More answers than the teacher• Less than the experienced researcher
Answers coded by PépiDiag
Answers Unanalyzed by PépiDiag
Answers Unanalyzed by 3 experts
218/266 (82%)
48/266 (18%)
24/266 (9%)
![Page 17: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/17.jpg)
Correct answers
No correct answer is analyzed incorrectly Unanalyzed answers
5/13 : Using X instead of * 8/13 : Mixing algebra and natural language :
• « (a+b) times (3+a) » Solutions
To prevent students from typing letters other than those relating to the exercise statement
17
Correct answers
Correct automatic coding
Unanalyzed answers by PépiDiag
Unanalyzed answersBy experts
149/266 136/149 13/149 0(56%) (91%) (9 %)
![Page 18: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/18.jpg)
Incorrect coding b+ axa +3 (parenthesis errors) b + a² + 3 (gathering the figure items) Both are incorrect (V3) distinct for human experts
and the same for the CAS Unanalyzed : incompleteness + interface
Incorrect answers
18
Incorrect answers
Correct automatic coding
Incorrect automaticcoding
Unanalyzed answers by PépiDiag
Unanalyzed answersBy experts
Unanalyzed by PépiDiagAnalyzed by experts
117/266 (44%)
80/82 2/82 35/117(30 %)
24/117(21 %)
11/117(9%)
![Page 19: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/19.jpg)
Conclusion Our approach to asses students’ open answers
Human experts• typify anticipated patterns of correct and
incorrect solutions • code each type on several dimensions
PépiDiag, using a CAS,• matches the student’s solution with a pattern
PépiDiag is reliable 100 % : closed answers 80 % : answers with one algebraic expression 70 % : answers with a multi-step reasoning [ITS
2008] PépiDiag saves teachers’ time and tedious effort
19
![Page 20: The Pépite project](https://reader035.vdocuments.net/reader035/viewer/2022062315/56816292550346895dd30372/html5/thumbnails/20.jpg)
Current work http://www.sesamath.net/ Diagnosis
To evaluate the building of the student’s profile from the answer assessment
Comparing• Heuristic method (implemented)• Statistical building (being implemented)• Neural Networks (being implemented)
A differentiation support tool for teachers To validate with teachers the different
learning paths adapted to each (group of profile) To build a tool to automatically
• retrieve indexed exercises from a data base • suggest a session adapted to a student’s profile 20