educational data sciences and the need for hermeneutic principles
DESCRIPTION
Educational Data Sciences and the Need For Hermeneutic Principles. An Interdisciplinary Perspective. Educational Data Sciences and the Need For Hermeneutic Principles Applied Epistemology . An Interdisciplinary Perspective. - PowerPoint PPT PresentationTRANSCRIPT
Educational Data Sciences and the Need For
Hermeneutic Principles
An Interdisciplinary Perspective
Educational Data Sciences and the Need For
Hermeneutic Principles Applied Epistemology
An Interdisciplinary Perspective
Educational Data Sciences and the Need For
Hermeneutic Principles Applied Epistemology
Interpretive Skills
An Interdisciplinary Perspective
Educational Data Sciences and the Need For Interpretive Skills
An Interdisciplinary Perspective
Some Driving Questions
• What counts as good learning analytics?• What kind of profession will data sciences
be?• What are its ancestor, sister, and adjoining
disciplines? • Which kinds of skills and dispositions are
important for preparing future practitioners and scholars?
Our Sociotechnical Thesis
• Data exist inside a social context; shaped by and shaping that context.
Our Sociotechnical Thesis
• Data exist inside a social context; shaped by and shaping that context.
• Interpretation is not technical. It is itself socially situated with goals, predispositions/ biases, and norms.
Our Sociotechnical Thesis
• Data exist inside a social context; shaped by and shaping that context.
• Interpretation is not technical. It is itself socially situated with goals, predispositions/ biases, and norms.
• Professional communities have developed valuable ways to reason from imperfect evidence. We can leverage/translate them to this new sociotechnical terrain.
Overview
1. Quantitative shifts in evidentiary artifacts (a digital ocean) in education
2. Qualitative shifts in educational focus
3. Some contributing/relevant disciplines
4. How to approach analysis, what kind of science/craft/skill/briciolage, etc. is it?
QUANTITATIVE AND QUALITATIVE SHIFTS IN EDUCATIONAL EVIDENCE
Computer
Adaptive testing
Assessment Technology
Computing Technology Central “Mainframe“
Computing
Personal Computing
Devices
Tabulating Technology
Cloud Technology
Services
Traditional fixed response, short task assessments
Analog Paper-based (Textbooks, worksheets, and manual classroom tools)
Analog Portfolio
Classroom Technology
The
Dig
ital
O
cean
Distributed Integrated Assessment
Systems
Dramatic Growth in Artifacts
Digital Classroom
Technology
1850s 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 20101850s 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010
The Digital Ocean• Test scores• Interim assessments• In class, formative assessments• Growth models• Student collaboration• Conversation records from
classroom talk and online tools • Student work, including rich and
multimodal demonstrations of knowledge and competency (essays, presentations, etc.)• Records of after-school
experiences• Records of informal learning • Activity traces from digital media
(in school, out of school, etc.)
• Demographics• Student-teacher relationships (TSDL) • School improvement plans/goals• Classifications (ex: proficiency
groups)• Video records of teaching• Annotated/evaluated records of
teaching• Teacher evaluations• Individual Education Plans (IEPs) and
personalized learning maps• Geospatial information
(mapping and trends)• Attendance and rosters (more
important than you think!)• FERPA/privacy blocks
Studying Oceans
Studying Oceans
Studying Oceans
Structures & Interrelationships
Diachronic/Change Processes
Variations in Affordance
Social Networks &Teams
Mobile Technology
Evidence and Transparency
Institution Focus
Teacher Control
Institutional Center Individual Student Nexus
Qualitative Shift in Emphasis
Related to the Educational Data Movement
SocialNetworksLearning
Networks
LearningCommuni
ties.
ExpertSources
Open Ed.Resources
Families
Qualitative Shift in Emphasis
WHAT DO STUDENTS KNOW?
Cognitive• Cognitive processes
and strategies• Knowledge• Creativity
Intrapersonal• Intellectual openness• Work ethic and
conscientiousness• Positive core self-
evaluation
Interpersonal• Teamwork and
collaboration• Leadership
Digi
tal M
edia
tion
• Critical thinking• Information literacy• Reasoning• Innovation
• Flexibility• Initiative• Appreciation for
diversity• Metacognition
• Communication• Collaboration• Responsibility• Conflict resolution
Artifacts
Qualitative Shift in Emphasis
Black Boxes Model
• Danny Hillis story; Oscon July 2012
Qualitative Shift in Emphasis
Black Boxes Model
• Danny Hillis story; Oscon July 2012
Explicit and Interrelated Components Model
Sociotechnical way of thinking about an educational system
THE DATA SCIENCES
Six Adjoining Disciplines
Educational Data Sciences
1. A new field with growing interest from leading universities, foundations, USED
2. Journals, conferences, and programs now emerging
3. What is the disciplinary focus, what counts as rigor and success?
EducationData
Sciences
Statistical Data Analysis
Statistical Data
Analysis
EducationData
Sciences
• Much of the digital ocean is compatible with statistical analysis.
• Exploratory data analysis (ex: Tukey with satellite data in 70s asked many questions that are being asked today about “big data”
• Already established (entrenched) in educational power structures
• Can produce strong claims
Learning Technology
Statistical Data
Analysis
EducationData
Sciences
Classroom/ Learning
Technology
• This is where the data we want most often come from…
• This area is seeing an explosion in media/learning resources and classroom management tools
Learning Sciences
Statistical Data
Analysis
EducationData
Sciences
Classroom/ Learning
Technology
Learning Sciences
• What does the data mean for multimodal, sociotechnical learning?
• How do socio-cultural and cognitive theories influence and be informed by data technologies?
• A design science for educational practice with iterative experiment, evaluate, refine process
Information Sciences
Statistical Data
Analysis
EducationData
Sciences
Classroom/ Learning
Technology
Learning Sciences
Information Sciences
• Visualizations and Human Computer Interface
• The information architectures that undergird data systems• Codes, classifications• Infrastructures and
boundary objects• Media centers and
educational resources
Organization/Management Sciences
Statistical Data
Analysis
Organization & Mgmt Sciences
EducationData
Sciences
Classroom/ Learning
Technology
Learning Sciences
Information Sciences
• Education is full of processes that can be designed
• Blended learning models are essentially re-structuring of organizational practices
• Inter-organizational functions are changing:• States-districts• Special education
Educational Data Sciences
Statistical Data
Analysis
Organization & Mgmt Sciences
EducationData
Sciences
Classroom/ Learning
Technology
Learning Sciences
Information Sciences
Decision Sciences
• Established field that uses large bodies of information to support organizational decisions
• As the volume and quality of educational data increase, more situations where decision sciences can be applied will emerge.
THE DATA SCIENCES
The Seventh and Generative Discipline
Educational Data Sciences
Statistical Data Analysis
Organization & Mgmt Sciences
Classroom/ Learning Technology
Learning Sciences
Information Sciences
Decision Sciences
Computer Science and EDM
Computer Science
Educational Data Sciences
Statistical Data Analysis
Organization & Mgmt Sciences
Classroom/ Learning Technology
Learning Sciences
Information Sciences
Decision Sciences
Computer Science and EDM
Computer Science
Educational Data Sciences
Statistical Data Analysis
Organization & Mgmt Sciences
Classroom/ Learning Technology
Learning Sciences
Information Sciences
Decision Sciences
Machine Learning
Data Mining
Hum-Comp. Interaction &Visualization
Natural Language Processing
Computational Statistics
Computer Science and EDM
REASONING FROM DIGITAL AGE EVIDENCE
Approaching Digital Age Data Analysis
• What counts as rigor and success?• Which parts of what disciplines are needed?• What methods are best?• What kinds of processes will make good,
great, and poor educational data analysts?• How much of the requirements are technical
versus attitudinal ?
Core Principles
1. All analytic processes are socially situated and iterative
2. Data is a mediational tool in an iterative process of discovery
3. Data is an imperfect lens for context and for interactions within that context
4. Organizational/systems thinking helps expand the reach of educational data science
5. Ethical as well as legal considerations are important.
Three Research Traditions
Evidence Centered
Design (ECD)
Exploratory Data Analysis
(EDA)
Linked Activity Systems
Framework (LASF)
Some Insights from ECD
• Arguments and Layers
• Domain Analysis: What is important in the domain?
• Domain Modeling: The Structure of Assessment Arguments
• Student, Evidence, and Task Models
Some Insights from CHAT/LASF
District Analytics
•Student Performance Data•Demographics •Rosters and assignments•Logs from technology tools
•Teacher and school characteristics•Home and demographic data
District Data Warehouse
•Census and geospatial data•Comparable data from SLDS
Teachers Teachers Teachers Teachers Teachers Teachers Teachers Teachers Teachers Elem.
Schools
Teachers Teachers Teachers Teachers Teachers Teachers Teachers Teachers
•Programmatic Evaluation•Quality of online tools•Professional Development•School Feedback
• Budgets and operating costs •Student and parent surveys
Traditional Model
Blended Model
Engeström Cultural Historical Activity Theory (CHAT)
Piety and Behrens Linked Activity Systems Framework
• Data analysis generally involves more than one activity system.
• The technology plays a linking/mediating role across contexts as well as within
• Framework allows for conceptualizing privacy/authorized access space
Some Insights from EDA
Underlying Heuristics (4Rs)• Revelation (graphics)• Residuals (models)
• Resistance /robustness
• Re-expression: scale
Broader Meaning• What do we “see” in the data?• What in our data fits/does not fit
with our emergent model?• Do we have a summary that is
not easily fooled by unusual distributions/examples
• How do the explanations we see apply more broadly.
What Kinds of Skills/Aptitudes?
• Broad fluency with a range of qualitative/quantitative methods
• Ethics, privacy, and confidentiality (FERPA+)• Technology accumen
The Educational Data Movement
• Systemic viewpoint: across silos and inter-organizational understanding
Though a famous and successful statistician, Tukey wanted to create a field that dealt with all data, even when it came in suchpoor shape that it was not amenable to statistical analysis. He called it “data analysis” and created the field called “ExploratoryData Analysis”. My undergraduate degree was in Psychology and Philosophy. I thought if I knew the logic of how we know things (epistemology) and understood the human lens through which all perception and thought occurs (psychology) I would have the fundamental layers of knowing from which to acquire more knowledge. After serving as a social worker and studying special education, I sought my Ph.D in Educational Psychology with a cognate called “Measurement, Statistics & Methodological Studies”. I would approach it as applied epistemology: How do we learn from data?
When I discovered Tukey’s writings I knew I had found the right place. I conducted psychological studies on perception ofstatistical graphics and wrote about the logical foundations of data analysis. When I wrote such a chapter called “Data and DataAnalysis” [6] people told me it was a silly title – data wasn’t a subject, it’s only a piece of the background to other sciences.Philosophy is concerned with understanding meaning and the application of logic. The philosopher asks What do we mean by‘data’? What do we mean by ‘analysis’? If data are symbols that point to elements in the world, what kind of logic do we need tounderstand that linkage? Like very good scientists, philosophers question the obvious. Such questioning may not be essential forwhat you do today, but it may open the door to do new ways of thinking you never imagined.
The successful learning analyst will avoid two common errors: Failure to understand the context and failure to become intimatelyfamiliar with the data. 1. The first error is caused by lack of contextual knowledge. Studying the learning sciences, education, and related disciplines will
help. 2. The second is error is caused by a substitution of complex statistical or computational models for detailed mental models. We
only build computational models or display to help our mental models.
Question the assumptions of your work deeply. It is important that analysts understand their work is about “revelation” or “unveiling” the reality of the world. It is a special (at times prophetic) role in society and should be taken very seriously.Do not think of data science as a set of techniques but as a collection of viewpoints (epistemic positions) and habits of mind.To undertake good visualization we need to know the techniques of data display, but also the psychology of perception, theanthropology of semiotics, the mathematics of fluctuation and the philosophy and art of aesthetic engagement. We will always need good technical analysts, but we need them to be
Drivers of Educational Data Mining
• Personalized Learning • College Going• Human Capital
Big Question
• What is the new role for the teacher?• Privacy • Getting designation as school vendors • Funder issues (12-24 months 2,3,5 yrs)
• Speak Gate-ish• Issue of interoperability…• Continuous Improvement from ECD
• Activity theory tensions• Activity systems in coherence/contrast• Innovations that help resolve Activity System
tensions (Engestrom, end of life care). • More data about the context