a metrics framework for evaluating group formation asma ounnas, david millard, hugh davis learning...
TRANSCRIPT
![Page 1: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/1.jpg)
A Metrics Framework for Evaluating Group FormationAsma Ounnas, David Millard, Hugh Davis
Learning Societies LabSchool of Electronics and Computer Science
The University of Southampton, UKwww.ecs.soton.ac.uk/~ao05r
![Page 2: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/2.jpg)
![Page 3: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/3.jpg)
A Metrics Framework for Evaluating Group FormationAsma Ounnas, David Millard, Hugh Davis
Learning Societies LabSchool of Electronics and Computer Science
The University of Southampton, UKwww.ecs.soton.ac.uk/~ao05r
![Page 4: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/4.jpg)
Group Formation
g1
g2
gN
Formation in terms of the constraints
+ Collaboration Goals
as a set of constraints
Students
Instructor
Groups
Software Engineering: I want all the students to have the opportunity to learn and perform well:
• No female can be allocated to an all-male group• No international student with all home-students• No groups should have participants from the same country (international)• distribute participants based on previous marks
• No minorities• Groups are to be multicultural• Groups are to be balanced in terms of expected performance
Constraints are dependent on
teachers’ pedagogy
![Page 5: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/5.jpg)
Constraint-based Group Formation
• The allocation of participants to groups based on some constraints
• Collaboration task has a set of goals, each is a set of constraints
• User has a degree of freedom in choosing the constraints
• Each constraint has a value
• Maximize the utility of all constraints within all goals => Optimal formation
![Page 6: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/6.jpg)
Assumptions
• Every student is a member of some group
• Non-overlapping group formation
• All groups have a similar size
• All formed groups are stable while the formation is not announced by the instructor
![Page 7: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/7.jpg)
Metrics for Group Formation
• Collaboration task t • Groups g• Collaboration Goals α• Constraints c• Participants p • Productivity Q(t)
• Cohort G• Formation form
{c1, c2}
{c1, c2 , cj}
{c1}
{c1, c3 …cL}
{g1, g2 .. gN}
{g1, g2 .. gN}
{g1, g2 .. gN}
{g1, g2 .. gN}
c1
c2
cj
cL
α1
α2
αk
αK
t
form1
form2
goal constraints groups outputs
![Page 8: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/8.jpg)
Metrics SummaryMetrics Participant Group Cohort
Formation
Constraint Satisfaction Quality
Perceived Formation Satisfaction
Goal Satisfaction
Goal Satisfaction
Formation Quality
Productivity
![Page 9: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/9.jpg)
Formation Metrics (1)
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 10: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/10.jpg)
Formation Metrics (1)
1. Constraint Satisfaction Quality
For a constraint c, how well was c satisfied?
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 11: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/11.jpg)
Formation Metrics (1)
1. Constraint Satisfaction QualityFor a constraint c, how well was c satisfied?
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
v if c is satisfied, 0 otherwise
• Group Constraint Satisfaction QualityHow well did all group g satisfy c
fg,c = {
![Page 12: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/12.jpg)
Formation Metrics (1)
1. Constraint Satisfaction Quality
For a constraint c, how well was c satisfied?
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
v if c is satisfied, 0 otherwise
• Group Constraint Satisfaction QualityHow well did all group g satisfy c
• Cohort Constraint Satisfaction QualityHow well did all the groups
satisfy c
fg,c = {
![Page 13: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/13.jpg)
Formation Metrics (2)
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 14: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/14.jpg)
Formation Metrics (2)
2. Perceived Formation SatisfactionHow well was the formation perceived – individual’s satisfaction with the allocation to groups (s)
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 15: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/15.jpg)
Formation Metrics (2)
2. Perceived Formation SatisfactionHow well was the formation perceived – individual’s satisfaction with the allocation to groups (s)
• Individual Formation SatisfactionNormalized measure from questionnaires.
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 16: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/16.jpg)
Formation Metrics (2)
2. Perceived Formation SatisfactionHow well was the formation perceived – individual’s satisfaction with the allocation to groups (s)
• Individual Formation SatisfactionNormalized measure from questionnaires.
• Group Formation SatisfactionIndividual satisfactions of all members of the group
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 17: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/17.jpg)
Formation Metrics (2)
2. Perceived Formation SatisfactionHow well was the formation perceived – individual’s satisfaction with the allocation to groups (s)
• Individual Formation SatisfactionNormalized measure from questionnaires.
• Group Formation SatisfactionIndividual satisfactions of all members of the group
• Cohort Formation SatisfactionIndividual satisfactions of all members of all groups
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 18: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/18.jpg)
Goal Satisfaction Metrics (1)
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 19: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/19.jpg)
Goal Satisfaction Metrics (1)
1. Goal Satisfaction QualityHow well did the groups satisfy a goal αk within the collaboration task t
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 20: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/20.jpg)
Goal Satisfaction Metrics (1)
1. Goal Satisfaction QualityHow well did the groups satisfy a goal αk within the collaboration task t
• Group Goal Satisfaction QualityHow well the students’ allocation to that group satisfied αk
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 21: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/21.jpg)
Goal Satisfaction Metrics (1)
1. Goal Satisfaction QualityHow well did the groups satisfy a goal αk within the collaboration task t
• Group Goal Satisfaction QualityHow well the students’ allocation to that group satisfied αk
• Cohort Goal SatisfactionHow well were all the groups
formed in terms of satisfying αk
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 22: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/22.jpg)
Goal Satisfaction Metrics (2)
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 23: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/23.jpg)
Goal Satisfaction Metrics (2)
2. Formation QualityHow well were the groups formed in terms of satisfying all the goals of the collaboration task t
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 24: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/24.jpg)
Goal Satisfaction Metrics (2)
2. Formation QualityHow well were the groups formed in terms of satisfying all the goals of the collaboration task t
• Group Formation QualityHow well was a group formed in terms of all goals.
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 25: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/25.jpg)
Goal Satisfaction Metrics (2)
2. Formation QualityHow well were the groups formed in terms of satisfying all the goals of the collaboration task t
• Group Formation QualityHow well was a group formed in terms of all goals.
• Cohort Formation QualityHow well was the cohort formed in terms of all the goals and therefore task t
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 26: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/26.jpg)
Productivity Metrics
1. Group Productivity Quality How well did the group achieve task t
Measure of the quality of the group outcome against an absolute scale defined by the instructor
AA++
Metrics P G C
Formation
Constraint Satis Quality
Perceived Form Satis
Goal Satisfaction Goal Satis
Form Quality
Productivity
![Page 27: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/27.jpg)
Optimal Formation
The optimal formation is the optimal cohort that can result from the set of goals, such that the formation quality metrics are maximized.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.QuickTime™ and a
TIFF (Uncompressed) decompressorare needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture. …..
form1 form2formn
g1
g2
gm
![Page 28: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/28.jpg)
Optimal FormationProcedure - Calculating group formation quality Complexity depends
on solvers algorithm
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
c1
α1
c2
cl
… αk
…
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture. …
Given task t, optimal formation formopt
for each formation form
for each group g in the cohort C for each goal α in the task t
return formopt
then formopt form
if cohort formation quality > optimal quality
calculate cohort formation quality
calculate group formation quality
calculate group goal satisfaction
for each constraint c in goal α calculate group constraint satisfaction
![Page 29: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/29.jpg)
Future Work
• Evaluating the Metrics Study
Questionnaires for data collections Questionnaires for perceived formation quality
Data Sample Software Engineering Groups (66 students) Programming Groups (27 students)
• Evaluating the constraints• Web-based group formation system
![Page 30: A Metrics Framework for Evaluating Group Formation Asma Ounnas, David Millard, Hugh Davis Learning Societies Lab School of Electronics and Computer Science](https://reader036.vdocuments.net/reader036/viewer/2022070306/55160454550346d46f8b5cbe/html5/thumbnails/30.jpg)
Thank you :-)
Questions?
Suggestions?
Comments?
More info?
www.ecs.soton.ac.uk/~ao05r