designing electronic performance support systems to facilitate learning psyc 512 october 20, 2005...
Post on 21-Dec-2015
212 views
TRANSCRIPT
Designing Electronic Designing Electronic Performance Support Systems Performance Support Systems
to Facilitate Learningto Facilitate LearningPSYC 512PSYC 512
October 20, 2005October 20, 2005
Christina AbbottChristina Abbott
CitationCitation
Van Schaik, P., Pearson, R & Barker, P., Van Schaik, P., Pearson, R & Barker, P., Designing electronic performance support Designing electronic performance support systems to facilitate learning. Isystems to facilitate learning. Innovations in nnovations in Education & Teaching InternationalEducation & Teaching International, 39(4), , 39(4), 289-306.289-306.
Why This ArticleWhy This Article
ShortShort Has some glaring problemsHas some glaring problems Relates to terms we knowRelates to terms we know Provides a decision-making tree for Provides a decision-making tree for
descriptive and inferential statistics.descriptive and inferential statistics.
What an EPSS Should Do?What an EPSS Should Do?
Shiak, Pearson, and Barker state that an EPSSShiak, Pearson, and Barker state that an EPSS11 facility designed for educational need should:facility designed for educational need should: Assess skill – fill in the gaps in performanceAssess skill – fill in the gaps in performance Act as a transfer agent – by using on-Act as a transfer agent – by using on-
demand mechanisms.demand mechanisms. Act as an augmentation aid – improve Act as an augmentation aid – improve
performance over and above natural abilityperformance over and above natural ability
11EPSS = Electronic Performance Support SystemEPSS = Electronic Performance Support System
Design of This EPSSDesign of This EPSS
Goal: To design a learning tool for psychology Goal: To design a learning tool for psychology students that aids in their understanding of various students that aids in their understanding of various statistical methods.statistical methods.
Step 1 – Needs AnalysisStep 1 – Needs Analysis Step 2 – Design and DevelopmentStep 2 – Design and Development
DatabaseDatabase Help SystemHelp System Advisor SystemAdvisor System Personal Area and NotesPersonal Area and Notes
Advisor SystemAdvisor System The Advisor System is a series of decision-making trees. The one below is for The Advisor System is a series of decision-making trees. The one below is for
descriptive statistics.descriptive statistics.
Advisor SystemAdvisor System
The decision-making tree below is for inferential statistics.The decision-making tree below is for inferential statistics.
PurposePurpose
There were three main purposes of the study:There were three main purposes of the study: Analyze students’ knowledge of concepts in Analyze students’ knowledge of concepts in
quantitative research methods. quantitative research methods. Establish students’ performance with Establish students’ performance with
system. system. Establish students’ overall acceptance of the Establish students’ overall acceptance of the
system as a learning aid. system as a learning aid.
HypothesisHypothesis
The researchers could have increased the The researchers could have increased the criticality of the study by including a criticality of the study by including a hypothesis for each of the above purposes. hypothesis for each of the above purposes.
For example:For example: Researchers could have developed a hypothesis for Researchers could have developed a hypothesis for
the performance of a group who did not use the the performance of a group who did not use the EPSS to answer knowledge questions (control EPSS to answer knowledge questions (control group) with a group that has full access to the group) with a group that has full access to the system.system.
Research DesignResearch Design
ParticipantsParticipants Sample of 89 sophomore psychology studentsSample of 89 sophomore psychology students Students had taken two research classes in their Students had taken two research classes in their
first year of studyfirst year of study Three parts to the researchThree parts to the research
Knowledge of Concepts in Quantitative Research Knowledge of Concepts in Quantitative Research MethodsMethods
Use of the EPSSUse of the EPSS System AcceptanceSystem Acceptance
Phase 1 - Knowledge of ConceptsPhase 1 - Knowledge of Concepts
Independent Variables – the test questions (nominal)Independent Variables – the test questions (nominal) Dependent Variables – the answers (ordinal)Dependent Variables – the answers (ordinal) ConfoundsConfounds
confidence level with computers and prior knowledge of confidence level with computers and prior knowledge of statistical methodsstatistical methods
There was no control group specified, i.e. one that There was no control group specified, i.e. one that answered the questions without using the EPSS.answered the questions without using the EPSS.
Could have made the study a within subjects design Could have made the study a within subjects design and measured their knowledge before and after using and measured their knowledge before and after using the EPSS.the EPSS.
Phase 2 – Use of the EPSSPhase 2 – Use of the EPSS
Independent Variables – the test questions (nominal)Independent Variables – the test questions (nominal) Dependent Variables – the answers (ordinal) (either Dependent Variables – the answers (ordinal) (either
correct or found)correct or found) ConfoundsConfounds
confidence level with computers and prior knowledge of confidence level with computers and prior knowledge of statistical methods were confounds. statistical methods were confounds.
there was no control group specified, i.e. one that answered there was no control group specified, i.e. one that answered the questions without using the Advisor, Help, and Tools. the questions without using the Advisor, Help, and Tools.
Could have made the study a within subjects design Could have made the study a within subjects design and measured their knowledge before and after using and measured their knowledge before and after using the tool.the tool.
Phase 3 – System AcceptancePhase 3 – System Acceptance Survey was a 7 point Lickert Scale with values ranging from Survey was a 7 point Lickert Scale with values ranging from
strongly agree to strongly disagree.strongly agree to strongly disagree. Independent Variables – the survey questions (nominal)Independent Variables – the survey questions (nominal) Dependent Variables – the answers (ordinal) Dependent Variables – the answers (ordinal) ConfoundsConfounds
The students success in Phase 1 and 2 may contribute to their overall The students success in Phase 1 and 2 may contribute to their overall system acceptance.system acceptance.
In previous studies I have done, when the user is asked to rate the In previous studies I have done, when the user is asked to rate the performance of a system, they are really rating his or her success with performance of a system, they are really rating his or her success with the system. These users had low expectations of the system to begin the system. These users had low expectations of the system to begin with and expected to struggle. Satisfaction level self-reports can be with and expected to struggle. Satisfaction level self-reports can be misleading. misleading.
There are more probing questions the researchers could have asked, There are more probing questions the researchers could have asked, such as “where did you get stuck” and “when were you lead astray by such as “where did you get stuck” and “when were you lead astray by the system.”the system.”
Hypothesis ImplicationsHypothesis Implications The null hypothesis was not stated at the beginning of the The null hypothesis was not stated at the beginning of the
study. study. Discussion Section Statements implied that the researchers, Discussion Section Statements implied that the researchers,
though misguided, may have had a hypothesis in mind.though misguided, may have had a hypothesis in mind. ““The significant correlations between acceptance measures The significant correlations between acceptance measures confirmedconfirmed
the association between intention of use and perceived usefulness the association between intention of use and perceived usefulness found in previous research.”found in previous research.”
The researchers also cited a correlation, but did not show the data to The researchers also cited a correlation, but did not show the data to back up their claim that:back up their claim that:
““Overall student’s performance using the EPSS was higher than Overall student’s performance using the EPSS was higher than unaided unaided performanceperformance [control group] on knowledge questions.” [control group] on knowledge questions.”
The researchers could have easily correlated the relationship between The researchers could have easily correlated the relationship between performance and confidence, as they had already gathered the data.performance and confidence, as they had already gathered the data.
ResultsResults There was scope for improving student’s There was scope for improving student’s
performance.performance. Students performance using the EPSS was high.Students performance using the EPSS was high. Students performance using the EPSS was higher Students performance using the EPSS was higher
than unaided performance on knowledge questions.than unaided performance on knowledge questions. Non-significant correlation between student’s Non-significant correlation between student’s
performance in terms of knowledge of concepts and performance in terms of knowledge of concepts and success in using the EPSS.success in using the EPSS.
Students expected that the EPSS would be useful to Students expected that the EPSS would be useful to them for learning, revision, and completing them for learning, revision, and completing assignments.assignments.
Parting ThoughtsParting Thoughts Ways to improve this study:Ways to improve this study:
State the hypothesis clearly.State the hypothesis clearly. Have multiple hypothesis to aid in the criticality of study.Have multiple hypothesis to aid in the criticality of study. Correlate performance with acceptance – this is more powerful than Correlate performance with acceptance – this is more powerful than
each alone.each alone.
Ways to improve the article:Ways to improve the article: Do not use the term - confirm.Do not use the term - confirm. State all aspects of the research design - include any control group State all aspects of the research design - include any control group
information.information.
Median measure used for Lickert Scale Data Median measure used for Lickert Scale Data