et4 online symposium, 2013 using text analytics to enhance data-driven decision making liz wallace...
DESCRIPTION
Connections All analyses and stakeholders are interrelatedTRANSCRIPT
ET4 Online Symposium, 2013
Using Text Analytics to Enhance Data-Driven Decision Making
Liz WallaceDirector, Institutional Research
Melissa Layne, Ed.D.Director, Research and Methodology
Phil Ice, Ed.D.VP, Research & Development
ASSESSMENT AT APUSA D3M Culture
ConnectionsAll analyses and stakeholders
are interrelated
Levels of AnalysisA range of approaches are required to
satisfy stakeholder needs
Descriptive Data
Inferential
Retention Learning Effectiveness Instructional DesignBeta Level Technology Integration
Regression Factor AnalysisDecision Trees
Predictive ModelingFederation of multiple demographic
and transactional data sets
Retention and Causality
Understanding who is likely to dis-enroll is different than why
Quantitative measures are not adequate for implementing systematic change across the enterprise
Explanatory data is needed
End of Course Survey Data
APUS participates in the Community ofInquiry (COI) End of Course survey.
The COI is a validated instrument based onthe research around Social, Cognitive, andTeaching Presence.
More than 500,000 learners have used this instrument and created a strong baseline forfurther research.
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105
COMMUNITY OF INQUIRY FRAMEWORK
• A process model of learning in online and blended educational environments
• Grounded in a collaborative constructivist view of higher education
• Assumes effective online learning requires the development of a community of learners that supports meaningful inquiry and deep learning
SOCIAL PRESENCE – ELEMENTS
• Affective expression (expressing emotion, self-projection)
• Open communication (learning climate, risk free expression)
• Group cohesion (group identity, collaboration)
COGNITIVE PRESENCE – ELEMENTS
• Triggering event (sense of puzzlement)
• Exploration (sharing information & ideas)
• Integration (connecting ideas)
• Resolution (synthesizing & applying new ideas)
TEACHING PRESENCE – ELEMENTS
• Design and organization (setting curriculum & activities)
• Facilitation (shaping constructive discourse)
• Direct instruction (focusing & resolving issues)
COMMUNITY OF INQUIRY SURVEY
• 9 social presence items (3 affective expression, 3 open communication, 3 group cohesion)
• 12 cognitive presence items (3 triggering, 3 exploration, 3 integration, 3 resolution)
• 13 teaching presence items (4 design & facilitation, 6 facilitation of discourse, 3 direct instruction)
COMMUNITY OF INQUIRY SURVEY - VALIDATION
• Tested in graduate courses at four institutions in the US and Canada
• Principal component factor analysis
• Three factor model predicted by CoI framework confirmed
• Arbaugh, Cleveland-Innes, Diaz, Garrison, Ice, Richardson, Shea & Swan – 2008
• Subsequent validation and cumulative n over 1 million – used at at least 50 institutions
Faculty EvaluationCombining descriptive, regression
and factor analysis
Anomalies
• Factor patterns can be an indication of problems
• Two factor solution found to produce lower level learning outcomes
• Four factor pattern associated with higher levels of disenrollment
• Factor analytics indicate that there is a problem, not what it is
Explanatory Data
• Understanding trends on end of course survey is useful but not adequate for implementing change
• Explanatory data must be utilized for clarification
• Utilization of qualitative survey data
• Large volume of data makes traditional qualitative methods impractical
• Exploration needs to go beyond branch / node analysis
• Text analytics utilized at APUS
Text Analysis• Utilized IBM/SPSS Text Analytics for Surveys
• Large volume of data - started with a 2-month sample to “train” the model
• Libraries existed in Text Analytics, but they were not specific to CoI or Higher Education
• Existing Opinions library helped identify Positive and Negative comments
• Needed to define CoI categories and identify the terms that were related to each category
• Decided to focus on Teaching Presence and Cognitive Presence
Teaching Presence
• Teaching presence • Design & Organization• Facilitation of Discourse• Direct instruction
• Used Community of Inquiry questions to begin to define the categories.
• Coded each category and a Positive (+) and a Negative (-) grouping for each
• Example: Direct Instruction, Direct Instruction-Positive, Direct Instruction-Negative
Teaching Presence
Cognitive Presence
• Cognitive presence items • Triggering Event• Exploration• Integration• Resolution
• Used Community of Inquiry questions to begin to define the categories.
• Coded each category and a Positive and a Negative grouping for each
• Example: Exploration, Exploration-Positive, Exploration-Negative
Cognitive Presence
Testing
• Tested/trained the model by having someone who knows the CoI well (Phil Ice & Melissa Layne) code samples and compared to the results from the model
• Repeated with new samples to improve accuracy
• Final test with new data sample (not used for training) to most-effectively measure accuracy
• Testing based upon 2011 data was approximately 80% accurate
Considerations
• Categories under Teaching Presence were more commonly and accurately assigned
• Fewer responses related to Triggering Event, Resolution and Integration than to the category of Exploration
Considerations
• Model will never be 100% accurate because of phrasing or sarcasm
• “I don’t think anything was exceptional during this class.” -- may be flagged as positive
• “That professor was so dynamic – as dynamic as an old shoe.” -- may be flagged as positive
• “Not exceptional” or “not dynamic” would be coded correctly
Application of Text Analysis
• Included courses identified as 3 or 4 factor courses
• Included only survey responses with comments
• Applied to surveys with at least one of the categories (such as Exploration) with a mean average score that fell below 3 (on a scale of 1-5)
• This data was loaded into the IBM/SPSS Text Analytics for Surveys model
• Results were exported to Excel
Exploration
• Random sampling of data with thematic extraction
• Validation through second round random pulls
• Consistency with conceptual framework of the CoI
• Does it pass the sniff test?
Examples
• Lack of clarity in instructional design and organization – bad directions
• No value given to discussion based activities
• Disconnect with group
• Triggering event poorly structured
• Lack of perceived applicability
ET4 Online Symposium, 2013
Thank You!Liz Wallace
Director, Institutional [email protected] Melissa Layne, Ed.D.
Director, Research and [email protected]
Phil Ice, Ed.D.VP, Research & Development