designing electronic performance support systems to facilitate learning psyc 512 october 20, 2005...

Designing Electronic Designing Electronic Performance Support Systems Performance Support Systems

to Facilitate Learningto Facilitate LearningPSYC 512PSYC 512

October 20, 2005October 20, 2005

Christina AbbottChristina Abbott

CitationCitation

Van Schaik, P., Pearson, R & Barker, P., Van Schaik, P., Pearson, R & Barker, P., Designing electronic performance support Designing electronic performance support systems to facilitate learning. Isystems to facilitate learning. Innovations in nnovations in Education & Teaching InternationalEducation & Teaching International, 39(4), , 39(4), 289-306.289-306.

Why This ArticleWhy This Article

ShortShort Has some glaring problemsHas some glaring problems Relates to terms we knowRelates to terms we know Provides a decision-making tree for Provides a decision-making tree for

descriptive and inferential statistics.descriptive and inferential statistics.

What an EPSS Should Do?What an EPSS Should Do?

Shiak, Pearson, and Barker state that an EPSSShiak, Pearson, and Barker state that an EPSS11 facility designed for educational need should:facility designed for educational need should: Assess skill – fill in the gaps in performanceAssess skill – fill in the gaps in performance Act as a transfer agent – by using on-Act as a transfer agent – by using on-

demand mechanisms.demand mechanisms. Act as an augmentation aid – improve Act as an augmentation aid – improve

performance over and above natural abilityperformance over and above natural ability

11EPSS = Electronic Performance Support SystemEPSS = Electronic Performance Support System

Design of This EPSSDesign of This EPSS

Goal: To design a learning tool for psychology Goal: To design a learning tool for psychology students that aids in their understanding of various students that aids in their understanding of various statistical methods.statistical methods.

Step 1 – Needs AnalysisStep 1 – Needs Analysis Step 2 – Design and DevelopmentStep 2 – Design and Development

DatabaseDatabase Help SystemHelp System Advisor SystemAdvisor System Personal Area and NotesPersonal Area and Notes

Advisor SystemAdvisor System The Advisor System is a series of decision-making trees. The one below is for The Advisor System is a series of decision-making trees. The one below is for

descriptive statistics.descriptive statistics.

Advisor SystemAdvisor System

The decision-making tree below is for inferential statistics.The decision-making tree below is for inferential statistics.

PurposePurpose

There were three main purposes of the study:There were three main purposes of the study: Analyze students’ knowledge of concepts in Analyze students’ knowledge of concepts in

quantitative research methods. quantitative research methods. Establish students’ performance with Establish students’ performance with

system. system. Establish students’ overall acceptance of the Establish students’ overall acceptance of the

system as a learning aid. system as a learning aid.

HypothesisHypothesis

The researchers could have increased the The researchers could have increased the criticality of the study by including a criticality of the study by including a hypothesis for each of the above purposes. hypothesis for each of the above purposes.

For example:For example: Researchers could have developed a hypothesis for Researchers could have developed a hypothesis for

the performance of a group who did not use the the performance of a group who did not use the EPSS to answer knowledge questions (control EPSS to answer knowledge questions (control group) with a group that has full access to the group) with a group that has full access to the system.system.

Research DesignResearch Design

ParticipantsParticipants Sample of 89 sophomore psychology studentsSample of 89 sophomore psychology students Students had taken two research classes in their Students had taken two research classes in their

first year of studyfirst year of study Three parts to the researchThree parts to the research

Knowledge of Concepts in Quantitative Research Knowledge of Concepts in Quantitative Research MethodsMethods

Use of the EPSSUse of the EPSS System AcceptanceSystem Acceptance

Phase 1 - Knowledge of ConceptsPhase 1 - Knowledge of Concepts

Independent Variables – the test questions (nominal)Independent Variables – the test questions (nominal) Dependent Variables – the answers (ordinal)Dependent Variables – the answers (ordinal) ConfoundsConfounds

confidence level with computers and prior knowledge of confidence level with computers and prior knowledge of statistical methodsstatistical methods

There was no control group specified, i.e. one that There was no control group specified, i.e. one that answered the questions without using the EPSS.answered the questions without using the EPSS.

Could have made the study a within subjects design Could have made the study a within subjects design and measured their knowledge before and after using and measured their knowledge before and after using the EPSS.the EPSS.

Phase 2 – Use of the EPSSPhase 2 – Use of the EPSS

Independent Variables – the test questions (nominal)Independent Variables – the test questions (nominal) Dependent Variables – the answers (ordinal) (either Dependent Variables – the answers (ordinal) (either

correct or found)correct or found) ConfoundsConfounds

confidence level with computers and prior knowledge of confidence level with computers and prior knowledge of statistical methods were confounds. statistical methods were confounds.

there was no control group specified, i.e. one that answered there was no control group specified, i.e. one that answered the questions without using the Advisor, Help, and Tools. the questions without using the Advisor, Help, and Tools.

Could have made the study a within subjects design Could have made the study a within subjects design and measured their knowledge before and after using and measured their knowledge before and after using the tool.the tool.

Phase 3 – System AcceptancePhase 3 – System Acceptance Survey was a 7 point Lickert Scale with values ranging from Survey was a 7 point Lickert Scale with values ranging from

strongly agree to strongly disagree.strongly agree to strongly disagree. Independent Variables – the survey questions (nominal)Independent Variables – the survey questions (nominal) Dependent Variables – the answers (ordinal) Dependent Variables – the answers (ordinal) ConfoundsConfounds

The students success in Phase 1 and 2 may contribute to their overall The students success in Phase 1 and 2 may contribute to their overall system acceptance.system acceptance.

In previous studies I have done, when the user is asked to rate the In previous studies I have done, when the user is asked to rate the performance of a system, they are really rating his or her success with performance of a system, they are really rating his or her success with the system. These users had low expectations of the system to begin the system. These users had low expectations of the system to begin with and expected to struggle. Satisfaction level self-reports can be with and expected to struggle. Satisfaction level self-reports can be misleading. misleading.

There are more probing questions the researchers could have asked, There are more probing questions the researchers could have asked, such as “where did you get stuck” and “when were you lead astray by such as “where did you get stuck” and “when were you lead astray by the system.”the system.”

Hypothesis ImplicationsHypothesis Implications The null hypothesis was not stated at the beginning of the The null hypothesis was not stated at the beginning of the

study.  study.  Discussion Section Statements implied that the researchers, Discussion Section Statements implied that the researchers,

though misguided, may have had a hypothesis in mind.though misguided, may have had a hypothesis in mind. ““The significant correlations between acceptance measures The significant correlations between acceptance measures confirmedconfirmed

the association between intention of use and perceived usefulness the association between intention of use and perceived usefulness found in previous research.”found in previous research.”

The researchers also cited a correlation, but did not show the data to The researchers also cited a correlation, but did not show the data to back up their claim that:back up their claim that:

““Overall student’s performance using the EPSS was higher than Overall student’s performance using the EPSS was higher than unaided unaided performanceperformance [control group] on knowledge questions.” [control group] on knowledge questions.”

The researchers could have easily correlated the relationship between The researchers could have easily correlated the relationship between performance and confidence, as they had already gathered the data.performance and confidence, as they had already gathered the data.

ResultsResults There was scope for improving student’s There was scope for improving student’s

performance.performance. Students performance using the EPSS was high.Students performance using the EPSS was high. Students performance using the EPSS was higher Students performance using the EPSS was higher

than unaided performance on knowledge questions.than unaided performance on knowledge questions. Non-significant correlation between student’s Non-significant correlation between student’s

performance in terms of knowledge of concepts and performance in terms of knowledge of concepts and success in using the EPSS.success in using the EPSS.

Students expected that the EPSS would be useful to Students expected that the EPSS would be useful to them for learning, revision, and completing them for learning, revision, and completing assignments.assignments.

Parting ThoughtsParting Thoughts Ways to improve this study:Ways to improve this study:

State the hypothesis clearly.State the hypothesis clearly. Have multiple hypothesis to aid in the criticality of study.Have multiple hypothesis to aid in the criticality of study. Correlate performance with acceptance – this is more powerful than Correlate performance with acceptance – this is more powerful than

each alone.each alone.

Ways to improve the article:Ways to improve the article: Do not use the term - confirm.Do not use the term - confirm. State all aspects of the research design - include any control group State all aspects of the research design - include any control group

information.information.

Median measure used for Lickert Scale Data Median measure used for Lickert Scale Data