tests & quizzes (aka samigo) keli amann, user experience specialist christine doherty, user...
TRANSCRIPT
Tests & Quizzes (aka SAMigo)
Keli Amann, User Experience SpecialistChristine Doherty, User Support Specialist
Stanford UniversityJune 17, 2010
• Feature Overview• 2.7 Highlights• 2.8 Roadmap• Assessment in 3akai• Q&A
Agenda
211th Sakai Conference - June 15-17, 2010
What are SAMigo’s Features:
311th Sakai Conference - June 15-17, 2010
• Assessment Authoring• Assessment Publishing• Assessment-Taking Management• Assessment Grading• Question Pools Management• Assessment-Type Management
• Online test• Self-study questions• Homework, problem-sets• Paper, code, or project submission• Language drills• Quick knowledge probes• Surveys• Others???
Pedagogical Flexibility
411th Sakai Conference - June 15-17, 2010
2.7 Highlights
11th Sakai Conference - June 15-17, 2010 6
Main Page
711th Sakai Conference - June 15-17, 2010
Assessments
Pending Assessments
Published Assignments
Active
Inactive
New AssessmentsCreate assessment ORImport assessment
11th Sakai Conference - June 15-17, 2010 9
2.62.6
2.72.7
Main Page: Create
11th Sakai Conference - June 15-17, 2010 10
2.62.6
2.72.7
Main Page: Create
11th Sakai Conference - June 15-17, 2010 11
2.62.6
2.72.7
Main Page: Create
Main Page, Pending
1211th Sakai Conference - June 15-17, 2010
Assessments
Pending Assessments
Published Assignments
Active
Inactive
New AssessmentsCreate assessment ORImport assessment
Main Page: Pending
11th Sakai Conference - June 15-17, 2010 14
2.62.6
2.72.7
Main Page: Published
1511th Sakai Conference - June 15-17, 2010
Assessments
Pending Assessments
Published Assignments
Active
Inactive
New AssessmentsCreate assessment ORImport assessment
Main Page: Published
11th Sakai Conference - June 15-17, 2010 17
2.62.6
2.72.7
*OOTB, Edit (not pictured) only appears if students haven’t started
Main Page: Columns
11th Sakai Conference - June 15-17, 2010 18
2.62.6
2.72.7
Edit Assessment Page
1911th Sakai Conference - June 15-17, 2010
Assessments
Pending Assessments
Published Assignments
Active
Inactive
New AssessmentsCreate assessment ORImport assessment
Edit Assessment
Assessment Settings
Publish
Preview
Auto-scored• Multiple Choice (single and multiple correct)• Survey (multiple choice, different rating scales available)• Fill in the Blank• Numeric Response• Matching• True/False
Manually scored• Short Answer/Essay• Audio Recording• File Upload
About Question Types
2011th Sakai Conference - June 15-17, 2010
11th Sakai Conference - June 15-17, 2010 22
2.62.6
2.72.7
Edit Assessment Page: Add Part
11th Sakai Conference - June 15-17, 2010 23
2.62.6
2.72.7
Edit Assessment Page: Preview, Publish
About Settings• Configures delivery of assessments (how they appear to
students)
• Options include: • display/organization• late handling• number of submissions • feedback/grading options • high security and time limit for high stakes testing
• Most settings are editable after publishing, but it is still important for instructors to review carefully
2411th Sakai Conference - June 15-17, 2010
Improvements to Preview
• Fill in the Blank and Numeric Response text fields and Matching selection menus made functional
• Timer (inaccurate in Preview mode) replaced with graphic/explanation
• Submit for Grading and Exit confirmation pages included
11th Sakai Conference - June 15-17, 2010 25
Preview screen
11th Sakai Conference - June 15-17, 2010 26
Partial Credit for Multiple Choice Questions
Allows instructor to enter percentages of credit to give student for selections other than correct answer
• Correct answer is set to 100% automatically; can not be edited
• Reset links allow instructor to go back to default grading logic or reset percentage values to 0
• Validation requires value to be between 0 and 100 (inclusive); no negative numbers allowed
• Applies to Single Correct Answer only
• Can not be used in conjunction with Negative Marking
• Feature needs to be turned on in Sakai Properties; not OOTB
• Contributed by Rice University
11th Sakai Conference - June 15-17, 2010 27
11th Sakai Conference - June 15-17, 2010 28
11th Sakai Conference - June 15-17, 2010 29
11th Sakai Conference - June 15-17, 2010 30
11th Sakai Conference - June 15-17, 2010 31
11th Sakai Conference - June 15-17, 2010 32
Print Assessment (with or without feedback)
• Print action appears in Actions menu on main authoring page and as link in Edit Assessment page
• Available for both pending and published assessments
• Options include:• Showing short answer key• Showing feedback (question-level and selection-level)• Showing intro and part titles• Changing font size of document• Printing as HTML or downloading a PDF
• Feature needs to be turned on in Sakai Properties; not OOTB • Contributed by Universidad Politecnica de Valencia
11th Sakai Conference - June 15-17, 2010 33
11th Sakai Conference - June 15-17, 2010 34
11th Sakai Conference - June 15-17, 2010 35
11th Sakai Conference - June 15-17, 2010 36
11th Sakai Conference - June 15-17, 2010 37
11th Sakai Conference - June 15-17, 2010 38
11th Sakai Conference - June 15-17, 2010 39
Email Notification• Option to send site members (or group members) email notification upon
publishing or republishing assessment
• Publishing without notification is set as the default to prevent accidental spamming
• Important settings are pre-populated in notification:• Assessment title• Available date, due date• Group released to (entire site, section(s)/group(s) or anonymous users) • Assessment URL• Time limit (if applicable)• Number of submissions allowed• Grading and feedback settings
• Option to include a personal message
11th Sakai Conference - June 15-17, 2010 40
11th Sakai Conference - June 15-17, 2010 41
11th Sakai Conference - June 15-17, 2010 42
11th Sakai Conference - June 15-17, 2010 43
11th Sakai Conference - June 15-17, 2010 44
Editable Parts in Published Assessments
• In 2.6, only questions and settings were editable in published assessments
• In 2.7, Parts are editable in the following ways:• Title• Information (longer text field)• Attachments (add or remove)
• Operations that are more likely to create discrepancies in scoring are not editable:• Random Draw options• Question Ordering (when not using Random Draw)
11th Sakai Conference - June 15-17, 2010 45
11th Sakai Conference - June 15-17, 2010 46
11th Sakai Conference - June 15-17, 2010 47
File Upload in Question Feedback
Option to upload files as feedback for each question; multiple files per question allowed
• Available on Questions page or student’s full submission page (under Scores)
• Update button must be clicked in scoring page after file uploaded with attachment helper
• Useful for returning commented papers to students
• Note: File size now displayed next to all file uploads (student submissions and files uploaded by instructor)
11th Sakai Conference - June 15-17, 2010 48
11th Sakai Conference - June 15-17, 2010 49
11th Sakai Conference - June 15-17, 2010 50
11th Sakai Conference - June 15-17, 2010 51
11th Sakai Conference - June 15-17, 2010 52
Improvements for No Submission Records
• Applies to Total Scores page when student identities can be seen by graders
• Comments: • Disabled for students with no submissions• ‘What’s This’ link to clarify purpose of Comments field
• Adjustment Scores:• If adjustment score is entered for student with no submission, the score is
now sent to the Gradebook and to the Export file• Useful for handling offline submissions (e.g., students experiencing technical
difficulties)
11th Sakai Conference - June 15-17, 2010 53
11th Sakai Conference - June 15-17, 2010 54
Improvements in Delivery Mode
• Changes in button labels• “Save and Continue” is now Next• “Save for Later” is now Exit
• Changes in button configuration• Navigation buttons to left, Save/Exit/Submit buttons to right• Submit for Grading is rightmost button on the last question page
• Auto-Save and Save (on demand)
• Improved button layout in audio recording applet
11th Sakai Conference - June 15-17, 2010 55
11th Sakai Conference - June 15-17, 2010 56
2.6 Button Configuration in Delivery Mode2.6 Button Configuration in Delivery Mode
11th Sakai Conference - June 15-17, 2010 57
2.7 Button Configuration in Delivery Mode2.7 Button Configuration in Delivery Mode
Auto-Save and Save (on demand)
• Auto-Save:• Saves student work without interruption (no screen refresh)• Runs at configurable interval (set in Sakai Properties)• Resets session timeout for inactivity• Useful for long assessments on one page or high stakes testing
• Save (on demand):• Allows students to save anytime while remaining on same page• Useful for answers that require significant time (e.g., essay questions)
• Contributed by Rutgers University
11th Sakai Conference - June 15-17, 2010 58
Improved Button Layout in Audio Recording Applet
• Record and Stop are two separate buttons now, replacing toggle • Reduces chances of students erasing recording by accident
11th Sakai Conference - June 15-17, 2010 59
Students View All Submissions/Scores
• In 2.6, only the submission with the recorded score (highest or last) displays in the student’s Submitted Assignments list
• In 2.7, the student’s Submitted Assignments list will show all submissions/scores by default
• Students can collapse the Submitted Assignments list to show only recorded scores
• Useful for self-study or practice exam scenarios; students can see feedback for multiple submissions
• Contributed by Universidad Politecnica de Valencia
11th Sakai Conference - June 15-17, 2010 60
11th Sakai Conference - June 15-17, 2010 61
11th Sakai Conference - June 15-17, 2010 62
Performance Improvements
• Performance has been improved in the following areas:
• Generation of Statistics for high number of submissions (~500)
• Loading of Questions page (under Scores) for File Upload and Audio Recording questions
• Downloading of student submission files (from File Upload questions)
11th Sakai Conference - June 15-17, 2010 63
2.8 Roadmap (independent)
64
• Respondus integration
• Record Average Score for multiple submissions*• Allow students to use Rich-Text Editor for Short Answer/Essay
question type**
• Remove 4000 character limit for student response in Short Answer/Essay question type**
• Allow instructors to enter zero score for unsubmitted assessments
• Allow instructors to show students scores only, without showing any other feedback (e.g., questions/responses)
• More to be discussed
* Based on contributions from Rice University** Based on contributions from Longsight
Assessment in Sakai 3.0?
65
User Research and Concept Development: Fall 09-Summer 2010
• No such thing as Tests & quizzes; functionality distributed
• All learning activities initiated from (syllabus schedule, to do, repository) in same basic way
• Activities stored in repository for sharing or reuse: find in “drawer” or repository
Work ongoing for
• instructor evaluation
• student views of syllabus, to do, grades
http://confluence.sakaiproject.org/display/UX/ Sakai+3.0+capabilities+for+learning+activities
Questions?
11th Sakai Conference - June 15-17, 2010 66
Contacts
Questions and feedback: [email protected]
Lydia Li, Stanford University Project Manager [email protected]
Homepage:http://confluence.sakaiproject.org/confluence/display/SAM
6711th Sakai Conference - June 15-17, 2010
Appendix
Review of capabilities of Sakai 2.x
6811th Sakai Conference - June 15-17, 2010
Organize & Publish Assessments
7011th Sakai Conference - June 15-17, 2010
• Pending Assessments• Starting page for assessment authoring• Can be created using markup text (aka Quick Create)• Can be exported/imported in QTI XML format• Can create duplicates easily • When ready, published via Settings
• Published Assessments for student assessment-taking• When students respond, Scores action appears• Can edit published assessments and most settings • Can remove published assessments
Auto-scored• Multiple Choice (single and multiple correct)• Survey (multiple choice, different rating scales available)• Fill in the Blank• Numeric Response• Matching• True/False
Manually scored• Short Answer/Essay• Audio Recording• File Upload
Question Types
7111th Sakai Conference - June 15-17, 2010
• At all levels of assessment authoring:• Assessment/Part Headers • Questions• Selections• Feedback
Rich Text & Attachments
7211th Sakai Conference - June 15-17, 2010
Quick Create
• Efficiency created by reducing number of clicks, pages loading and saving
• Useful for transferring assessments created in other applications or LMS (e.g., MS Word, WebCT), no need to paste into different fields
• Supports Multiple Choice, Fill in the Blank, Short Answer/Essay, and True/False question types
• Rich text, attachments, and additional questions can be added later, through assessment authoring page
7311th Sakai Conference - June 15-17, 2010
Import/Export
• Supports QTI v 1.2
• Export types: IMS QTI-compliant XML or IMS Content Packaging (for use with attachments)
• Useful for:• Copying assessment to different site• Archiving assessments offline
7411th Sakai Conference - June 15-17, 2010
Settings• Configures delivery of assessments (how they appear to
students)
• Options include: • display/organization• late handling• number of submissions • feedback/grading options • high security and time limit for high stakes testing
• Most settings are editable after publishing, but it is still important for instructors to review carefully
7511th Sakai Conference - June 15-17, 2010
Publishing• Release assessment to students
• Copy made from pending assessment – pending assessment can be modified without affecting published version
• Can set Available Date in future -- useful for instructors who want to do bulk of setup early
• Can retract any time, making assessment unavailable to all students
• Can remove published assessment
• Can edit published assessment
7611th Sakai Conference - June 15-17, 2010
Taking an Assessment (Students)
• Take an Assessment section• List determined by available, due, and retract dates,
plus number of allowed submissions• Can submit when done or save a work in progress
• Submitted Assessments section• Provides submission confirmation with timestamp• Depending on settings, students can review scores,
feedback, grader’s comments, statistics
7711th Sakai Conference - June 15-17, 2010
Grading
• Ability to grade all questions for one student or grade all responses to one question
• Manual score adjustments can be made to individual questions or entire assessment
• Comments can be made for individual questions or entire assessment
• Ability to send scores to Gradebook
• Ability to export scores/responses to Excel
• Ability to calculate statistics and histograms
7811th Sakai Conference - June 15-17, 2010
Question Pools
• Organize questions into multiple subpools
• Move/copy/remove multiple questions at a time
• Author questions from within Pool Manager
• Import questions from IMS QTI-compliant XML file
• Randomly draw from pools for an assessment
• Question pools belong to an individual user, available across all user’s sites
• Ability to share question pools with other site members
7911th Sakai Conference - June 15-17, 2010
Assessment Types
• Create assessment “types” for different disciplines, different uses
• Hide complexity by selecting which settings are visible to the instructor.
• Pre-set defaults by making setting choices for the type (e.g., Self-study always has immediate feedback)
8011th Sakai Conference - June 15-17, 2010
Assessment Creation Workflow
July 2009 10th Sakai Conference - Boston, MA, U.S.A.
11th Sakai Conference - June 15-17, 2010 82