a student measure of the quality of open digital textbooks
DESCRIPTION
Presentation to be given at OpenEd 2012 in Vancouver, BC.TRANSCRIPT
A Student Measure of the Quality of Open Digital Textbooks
OrA Method For Getting Data-Backed
Student Opinions
TJ Bliss
This work is licensed under a Creative Commons Attribution 3.0 Unported License. All credited images fall under Fair Use, are intended for transformative purposes only,
and retain full copyright privileges.
Education is necessary to the individual “to prepare him for the future life means to give him command of himself; it means so
to train him that he will have the full and ready use of all his capacities.”
Image Credit: http://c250.columbia.edu/
John Dewey
Image Credit: http://helenedison.ucsd.edu/
“Serving society is only one of higher education’s functions, but it is surely among the most important. At a time when the nation has its full share of difficulties...the question is not whether universities need to concern themselves with society’s problems but whether they are discharging this responsibility as well as they should.”
Derek Bok
Image Credit: http://thethinkerblog.com/
Image Credit: http://thethinkerblog.com/
Image Credit: http://edudemic.com/
Image Credits: flatworldknowledge.com; www.etftrends.com; www.eukicks.com
Image Credits: www.flatworldknowledge.com; www.mindingthecampus.com; www.debatepolitics.com
Dilbert By Scott Adams
Image credit: www.dilbert.com
It’s Alive! The Birth of a Textbook Quality Measure
Hard data to back up student opinions about the quality of their (open) digital textbooks
Image Credit: www.yankeeroundtable.com
“Final” Questionnaire
student surveys
student interviews
literature reviewQualitative Data
thematic analysis
Target Themes
mental modeling
Conceptual Model
item development
Pilot Questionnaire
factor analysis
Validated ModelValidated Model
“Final” Questionnaire
factor analysis
item revisionitem revision
Final Questionnaire
student surveys
student interviews
literature reviewQualitative Data
thematic analysis
Target Themes
mental modeling
Conceptual Model
item development
Pilot Questionnaire
factor analysis
Validated ModelValidated Model
“Final” Questionnaire
factor analysis
item revisionitem revision
Project Kaleidoscope
8 community colleges and 4-year schoolsCalifornia to New York
Project Kaleidoscope
Cross-institutional faculty teamsAggregate OER-based textbook replacements (FWK)
11 courses, 9,000 students in 2011-2012
490 Student Survey Responses
From your perspective, what are the characteristics of a high quality digital textbook?
Overall, what did you think of the digital textbook you used in your course?
How would you rate the quality of the digital textbook you used in your course? Why?
10 Student Interviews
In-depth probingClarifying questioning
Transcriptions
70 Journal Articles
Peer reviewed9 main textbook evaluation criteria
Final Questionnaire
student surveys
student interviews
literature reviewQualitative Data
thematic analysis
Target Themes
mental modeling
Conceptual Model
item development
Pilot Questionnaire
factor analysis
Validated ModelValidated Model
“Final” Questionnaire
factor analysis
item revisionitem revision
Thematic Analysis
7 main stepsData prep, reading, sorting, writing, axial coding,
more sorting, finalizing
Target Themes
Navigation, Access, Performance, Interaction, Relevance, Presentation, Impact, Diversity
Final Questionnaire
student surveys
student interviews
literature reviewQualitative Data
thematic analysis
Target Themes
mental modeling
Conceptual Model
item development
Pilot Questionnaire
factor analysis
Validated ModelValidated Model
“Final” Questionnaire
factor analysis
item revisionitem revision
Mental Modeling
Identify construct relationships
Final Questionnaire
student surveys
student interviews
literature reviewQualitative Data
thematic analysis
Target Themes
mental modeling
Conceptual Model
item development
Pilot Questionnaire
factor analysis
Validated ModelValidated Model
“Final” Questionnaire
factor analysis
item revisionitem revision
Item Development
WritingExpert review
RevisionCognitive interviews
RevisionExpert review
Revision
Pilot Questionnaire
6 Background Items44 Main Items
8 Target ThemesOctober 2012 Administration
Final Questionnaire
student surveys
student interviews
literature reviewQualitative Data
thematic analysis
Target Themes
mental modeling
Conceptual Model
item development
Pilot Questionnaire
factor analysis
Validated ModelValidated Model
“Final” Questionnaire
factor analysis
item revisionitem revision
Next Steps
Model evaluation and validation (Nov. ‘12)Item revision (Jan. ‘13)
“Final” Questionnaire 2.0 (Feb. ‘13)Further evaluation, validation, revision (Apr. ‘13)
“Final” Questionnaire 3.0 (May ‘13)