professional learning in informal online … · professional learning in informal online...
TRANSCRIPT
PROFESSIONAL LEARNING IN INFORMAL ONLINE ENVIRONMENTS: EXAMINING
ELEMENTARY TEACHERS’ COGNITIVE PROCESSES AND LEARNING EXPERIENCES
AS THEY NAVIGATE A PROFESSIONAL DEVELOPMENT WEBSITE
By
Pamela-Jane Kathleen Beach
A thesis submitted in conformity with the requirements
for the degree of Doctor of Philosophy
Graduate Department of Applied Psychology and Human Development
Ontario Institute for Studies in Education of the
University of Toronto
© Copyright by Pamela-Jane Kathleen Beach 2015
ii
PROFESSIONAL LEARNING IN INFORMAL ONLINE ENVIRONMENTS:
EXAMINING ELEMENTARY TEACHERS’ COGNITIVE PROCESSES AND
LEARNING EXPERIENCES AS THEY NAVIGATE A PROFESSIONAL
DEVELOPMENT WEBSITE
Doctor of Philosophy, 2015
Pamela-Jane Kathleen Beach
Department of Applied Psychology and Human Development
University of Toronto
Abstract
This thesis is comprised of two studies. The first is a mixed methods study comparing three
types of think aloud methods for examining elementary teachers’ cognitive processes as they
use a multimedia professional development website. Forty-five elementary teachers were
assigned to one of three think aloud conditions—concurrent, retrospective, and virtual revisit.
Participants in the concurrent condition verbalized their thoughts while simultaneously
navigating the website for 20 minutes; participants in the retrospective condition verbalized their
thoughts immediately following their 20-mintue navigation without any aids; and participants in
the virtual revisit condition verbalized their thoughts while viewing a screen recording of their
20-minute navigation. Results revealed differences (p < .05) in the complexity of verbalizations
produced by participants in each condition, in task completion, and in reported comfort levels
with thinking aloud.
The findings of study 1 revealed that participants from the virtual revisit condition
produced extensive complex verbalizations reflecting higher cognitive thought processes.
Therefore, a second study was conducted using qualitative methods. Specifically, a grounded
theory approach to analysis provided an in-depth understanding of the learning experiences of
participants from the virtual revisit condition as they navigated the given website. A theoretical
model was developed describing: (1) causal conditions that affect elementary teachers’
navigation of a professional development website, (2) the central phenomenon, navigating a
iii
professional development website, (3) navigational strategies, and (4) potential learning
outcomes.
Overall, the findings of this thesis provide insight into the benefits and limitations of
each think aloud method and of elementary teachers’ cognitive processes and learning
experiences during informal online professional learning. The results also highlight the need for
further investigations of teacher cognition and online learning using the virtual revisit think
aloud method. Educational and research implications are discussed for both studies.
iv
Acknowledgements
I would like to express my sincere gratitude to my thesis supervisor, Dr. Dale Willows
for the endless encouragement, guidance, and advice. She contributed immensely to this
thesis—by conceiving significant aspects of the virtual revisit think aloud, as well as providing
insightful comments and questions to refine my thoughts throughout the entire process. It has
been an honour to work under Dr. Willows’ guidance. Her dedication to teaching, learning, and
knowledge mobilization is inspiring.
I would like to thank my committee members, Dr. Rhonda Martinussen and Dr. Joan
Peskin, for their commitment, support, and valuable feedback during the entire process. I would
also like to express my gratitude to Dr. Vera Woloshyn for her willingness to undertake the role
of the external examiner, and for her insightful comments and questions. As well, I would like
to thank Dr. Janette Pelletier for her time and thoughtfulness.
I am indebted to Ronna Kluger and Robin Bennett for their ongoing support,
encouragement, and assistance during data collection. I would like to thank Jayme Herman for
her role as the second coder during the initial stages of coding, and for her insights and advice
during analysis. I am thankful to Dr. Yiola Cleovoulou for her continued encouragement,
guidance, and for sharing her knowledge of qualitative research with me. I would also like to
thank Dr. Eunice Jang and Dr. Olesya Falenchuk—the knowledge that I gained from their
research methods courses was instrumental in carrying out my analyses.
I would like to acknowledge the incredible Balanced Literacy Diet website team for their
dedication to the development of an amazing online teacher resource. Your support and
friendship are invaluable. Thank you to Madison Aitken, Kate Bryant, Lesley Dookie, Julia
Forgie, Kasia Kania, Nathalie Rothschild, and Taj Uppal.
v
Thank you to all of my friends and family for their positive energy, interest, and ongoing
support. I am especially thankful to my parents, Ross and Jane Beach for their enthusiasm,
endless encouragement, and occasional editing. And to Dani Oore—thank you for listening to
me think aloud about my research, for your thoughtfulness, support, and kind heart.
Finally, I would like to thank the teachers who participated in this research. Their
willingness to share their thoughts and experiences contributed to the understanding of teacher
learning.
vi
Table of Contents
Abstract ............................................................................................................................................ ii
Acknowledgements......................................................................................................................... iv
Table of Contents ............................................................................................................................ vi
List of Tables ................................................................................................................................... x
List of Figures ................................................................................................................................. xi
List of Appendices ......................................................................................................................... xii
CHAPTER 1—General Introduction .......................................................................................... 1
Introduction to the Studies ..................................................................................................... 1
Thesis Overview ..................................................................................................................... 6
CHAPTER 2—General Methodology.......................................................................................... 8
Context ................................................................................................................................... 8
Participant Recruitment ........................................................................................................ 11
Data Sources ......................................................................................................................... 12
Demographic Questionnaire ............................................................................................ 12
Think Aloud ..................................................................................................................... 15
Screen Capture Technology ............................................................................................. 16
Semi-structured Interview ............................................................................................... 16
Procedure .............................................................................................................................. 16
One-on-One Meeting ....................................................................................................... 16
CHAPTER 3—Study 1: Comparing Three Think Aloud Methods for Examining
Teachers’ Cognitive Processes as They Use a Multimedia Professional Development
Website.......................................................................................................................................... 19
Literature Review ................................................................................................................. 19
Think Aloud Methodology .............................................................................................. 19
Teacher Cognition ........................................................................................................... 23
Research Questions .............................................................................................................. 25
Methods ................................................................................................................................ 25
Research Design .............................................................................................................. 25
Data Analysis ................................................................................................................... 28
Data Review Process ....................................................................................................... 38
Results and Discussion ......................................................................................................... 38
vii
Research Question 1: To What Extent Do Participants’ Verbalizations Differ Across the
Three Think Aloud Methods? .......................................................................................... 39
Summary of Results and Discussion for Research Question 1........................................ 52
Research Question 2: How Do Think Aloud Tasks Differ Across the Three Think Aloud
Conditions? ...................................................................................................................... 52
Summary of Results and Discussion for Research Question 2........................................ 60
Research Question 3: To What Extent Does Comfort Level with Thinking Aloud Differ
Across the Three Think Aloud Methods? ........................................................................ 61
Summary of Results and Discussion for Research Question 3........................................ 66
General Discussion of Study 1 ............................................................................................. 67
A Participant’s Experience with the Concurrent Think Aloud Method .......................... 68
A Participant’s Experience with the Retrospective and Virtual Revisit Website Task ... 70
A Participant’s Experience with the Retrospective Think Aloud .................................... 71
A Participant’s Experience with the Virtual Revisit Think Aloud .................................. 71
Benefits and Limitations of Employing Each Type of Think Aloud Method ................. 72
Study 1 Limitations ......................................................................................................... 75
Significance and Educational Implications ..................................................................... 76
Conclusion ............................................................................................................................ 78
CHAPTER 4—Understanding Teachers’ Learning Experiences as They Use a Professional
Development Website: A Qualitative Study .............................................................................. 79
Theoretical Models for Understanding Teacher Learning ................................................... 81
Knowles’ Theory of Adult Learning ............................................................................... 81
Self-Directed Learning Theory ........................................................................................ 84
Shulman and Shulman’s Model of Teacher Learning ..................................................... 86
Formal and Informal Professional Development ................................................................. 88
Study Purpose and Research Questions ............................................................................... 91
Methods ................................................................................................................................ 92
Participants ...................................................................................................................... 92
Research Design .............................................................................................................. 94
Data Analysis ................................................................................................................... 96
Establishing Trustworthiness ......................................................................................... 101
Findings and Discussion ..................................................................................................... 105
Conditions Affecting Elementary Teachers’ Website Navigation ..................................... 108
Theme 1: Perception of Professional Learning .................................................................. 108
viii
1. Quality of Professional Resources ............................................................................. 108
2. Beliefs about Literacy Instruction ............................................................................. 112
Theme 2: Focusing on Student Needs and Instructional Goals.......................................... 114
1. Student Needs ............................................................................................................ 115
2. Classroom Context .................................................................................................... 115
3. Current and Future Instructional Goals ..................................................................... 116
Theme 3: Individual Differences ........................................................................................ 116
1. Motivating Factors to Use Online Resources ............................................................ 117
2. Online User Behaviour .............................................................................................. 119
3. Prior Experience and Professional Knowledge ......................................................... 122
The Central Phenomenon Resulting from the Conditions.................................................. 124
Theme 4: Navigating a professional development website ................................................ 124
1. Gaining a General Sense of the Website ................................................................... 125
2. Exploring In-Depth .................................................................................................... 125
3. Understanding the Website ........................................................................................ 126
Strategies for Navigating a Professional Development Website........................................ 127
Theme 5: Evaluating Information. ..................................................................................... 128
1. Content Richness ....................................................................................................... 129
2. Functionality of the Technology ................................................................................ 129
3. Extent of Technology Integration .............................................................................. 130
4. Novelty of the Technology and Uniqueness of the Learning Environment .............. 131
5. Potential for Learning ................................................................................................ 131
Theme 6: Encoding Information for Future Retrieval........................................................ 132
1. Note-Taking ............................................................................................................... 132
2. Saving Ideas for Future Use ...................................................................................... 133
Potential Outcomes Resulting from Elementary Teachers’ Website Navigation .............. 134
Theme 7: Reflecting on Learning ....................................................................................... 134
1. General Feelings about the Website .......................................................................... 135
2. Future Use of the Website ......................................................................................... 135
Theme 8: Continued Professional Learning ....................................................................... 136
1. Advantages of Online Professional Learning ............................................................ 137
2. Increased Self-Efficacy .............................................................................................. 137
3. Feeling Inspired ......................................................................................................... 139
ix
4. Knowledge Construction ........................................................................................... 140
Theme 9: Intentions for Practice. ....................................................................................... 140
1. Adapting Information ................................................................................................ 141
2. Professional Goals ..................................................................................................... 142
General Discussion ............................................................................................................. 143
Overview of Major Findings .............................................................................................. 144
Key Conditions .............................................................................................................. 144
Key Navigational Strategies .......................................................................................... 145
Key Potential Outcomes ................................................................................................ 145
Study 2 Limitations ............................................................................................................ 146
Significance and Future Directions .................................................................................... 148
CHAPTER 5—General Conclusions ....................................................................................... 151
References.................................................................................................................................... 153
Appendices .................................................................................................................................. 169
x
List of Tables
Table 2.1 Demographic Characteristics………………………………..………………….…….14
Table 3.1 Average Word Counts across the Three Conditions………………………………….29
Table 3.2 Coding Scheme-Categories Used to Code the Transcripts…………………….……..30
Table 3.3 Average Number of Thought Units across the Three Conditions……………………33
Table 3.4 Frequency of Thought Units across Conditions…………………….…….………….34
Table 3.5 Time Sampling Analysis-Variables………………………….……………………….36
Table 3.6 ANOVA Summary Statistics for Thought Units………………….……….………....40
Table 3.7 ANOVA Summary Statistics for Word Count and Number of Thought Units……....48
Table 3.8 Pearson Correlations…………………….…………………………………………....51
Table 3.9 ANOVA Summary Statistics for Web-Based Action Variables……………………..54
Table 3.10 Themes Related to Reported Comfort Level………………………………………..63
Table 3.11 Benefits and Limitations of Employing Each Type of Think Aloud Method……....74
Table 4.1 Demographic Characteristics for Study 2…………………………………………….93
Table 4.2 Study 2 Themes……………………………………………………………………..105
Table 4.3 General Agreement of What Constitutes Best Practices in Literacy Instruction……114
Table 4.4 Song and Lee’s Criteria for Evaluating Informal Online Environments……………128
xi
List of Figures
Figure 2.1 Homepage of The Balanced Literacy Diet website
Figure 3.1 Convergence mixed methods research design
Figure 3.2 Topics pertaining to research question 1 results and discussion
Figure 3.3 Topics pertaining to research question 2 results and discussion
Figure 3.4 Topics pertaining to research question 3 results and discussion
Figure 3.5 Topics pertaining to the general discussion of study 1
Figure 4.1 Outline of overall process for study 2
Figure 4.2 Topics pertaining to establishing trustworthiness
Figure 4.3 Theoretical model for elementary teachers’ learning experiences as they navigated a
professional development literacy website
Figure 4.4 Topics pertaining to theme 1: Perception of professional learning
Figure 4.5 Topics pertaining to theme 2: Focusing on student needs and instructional goals
Figure 4.6 Topics pertaining to theme 3: Individual differences
Figure 4.7 Topics pertaining to theme 4: Navigating a professional development website
Figure 4.8 Topics pertaining to theme 5: Evaluating information
Figure 4.9 Topics pertaining to theme 6: Encoding information for future retrieval
Figure 4.10 Topics pertaining to theme 7: Reflecting on learning
Figure 4.11 Topics pertaining to theme 8: Continued professional learning
Figure 4.12 Topics pertaining to theme 9: Intentions for practice
xii
List of Appendices
Appendix A-1 Sample Pages of The Balanced Literacy Diet Website
Appendix A-2 Invitation to Participate Script
Appendix A-3 Information Letter and Consent Form
Appendix A-4 Recruitment Poster
Appendix A-5 Demographic Characteristics across the Three Different Modes of Recruitment
Appendix A-6 Demographic Questionnaire
Appendix A-7 Think Aloud and Interview Consent Form
Appendix A-8 Screen Shot of Camtasia Studio During a Participant’s Navigation
Appendix A-9 Interview Questions
Appendix B-1 Initial List of Potential Categories for Coding Think Aloud Transcripts
Appendix B-2 Krathwohl’s Revised Taxonomy
Appendix B-3 Frequency Counts of Web-based Actions
Appendix B-4 Sample Observations of Web-based Actions with Think Aloud Recordings
Appendix B-5 Data Screening Results and Descriptives
Appendix C-1 Initial List of Categories for Study 2
Appendix C-2 Sample of Memos Written During Analysis
Appendix C-3 Themes and Descriptions Sent to Participants Involved in Member Check
Appendix C-4 Participants’ ‘Member Check’ Comments
Appendix C-5 Frequency of Themes across Data Sources and across Participants
Appendix C-6 Samples from Audit Trail
1
CHAPTER 1
General Introduction
This thesis presents two studies that investigate the cognitive processes and learning
experiences of elementary teachers as they use a multimedia professional development website.
The first investigation reports on the results of a mixed methods study that compares the relative
utility of three think aloud methods for examining elementary teachers’ cognitive processes as
they use a professional development website. Based on the findings of study 1, the second study
involves an in-depth investigation of elementary teachers’ online professional learning
experiences. Below is a brief introduction to the studies.
Introduction to the Studies
Elementary teachers are a necessary foundation for building successful programs in the
classroom (Gambrell & Anders Mazzoni, 1999; Pressley, Mohan, Raphael, & Fingeret, 2007).
Successful programs begin with a repertoire of pedagogical knowledge, pedagogical content
knowledge, and research-based instructional practices. This repertoire of information can be
delivered to practicing elementary teachers through various professional development
opportunities (Cervetti, Kulikowich, Drummond, & Billman, 2012; Desimone, 2009; Kao, Wu,
& Tsai, 2011).
One facet of teacher professional development is online learning, which occurs when
professional knowledge is constructed from multiple modes of digital information—
photographs, videos, and interactive tools, to name a few (Mayer, 2002). Online learning is a
favoured approach to professional development because it creates accessible opportunities;
online learning takes place within platforms that deliver information in a means that removes
time, place, and situational barriers (Kanuka & Nocente, 2003). As elementary teachers
increasingly turn to the Internet for their professional learning (Charalambousa & Ioannou,
2
2011; Kao, Wu, & Tsai, 2011), it is essential to examine how they use and learn from online
resources and professional development websites.
Teacher learning is a complex process that involves a range of cognitive activities,
decision-making strategies, and learning experiences. Studying teachers’ cognitive processes
and learning experiences in the context of online professional development has several
educational and research implications. The study of how teachers use online learning
environments can provide educational website developers with guidance in the design and
framework of professional development tools and technologies. Study outcomes can inform
educational website developers and administrators of professional development programs about
elementary teachers’ motivations to use professional development websites, their beliefs and
attitudes towards these learning platforms, and potential learning outcomes. In addition,
studying teacher online learning can lead to greater insight into teachers’ experiences within
these online environments. Greater insight into teachers’ cognitive processes and online
learning experiences can contribute to a holistic and multifaceted view of teacher professional
development.
Most studies that have examined online teacher learning have gathered data through
surveys, questionnaires, and interviews (Duncan-Howell, 2010; Hur & Brush, 2009; Kao et al.,
2011). These methods offer information about teachers’ attitudes towards online professional
learning; however, data generated from these methods is limited to participants’ recollection of
past events. A method that tracks teachers’ cognitive processes as they make online choices is
necessary to provide further insight into how teachers use and learn from online environments.
The think aloud methodology is an approach that can track teachers’ cognitive processes during
decision-making activities.
3
The think aloud methodology is based on the techniques of protocol analysis by Ericsson
and Simon (1984, 1993) and has been shown to provide direct data about the reasoning process
(van Someren, Barnard, & Sandberg, 1994). Verbal reports provide information about
experiences and knowledge that cannot easily be accessed by more traditional observations and
interviews (Ericsson & Simon, 1993). Two variations of the think aloud most often employed
by researchers are the concurrent think aloud and the retrospective report. During the
concurrent think aloud participants complete a task while they simultaneously verbalize their
thoughts. The retrospective report requires participants to verbalize their thoughts after they
complete a given task. Studies that have successfully employed the concurrent think aloud
method in the context of online learning have focused on how undergraduate university students
and elementary school students use and evaluate educational websites (Armstrong, 2011;
Barzilai & Zohar, 2012; Damico & Balidon, 2007); however, few studies that employ the
concurrent think aloud method focus on elementary teacher participants. The retrospective
report on its own has received little attention in the field of online learning, and research
comparing the concurrent and retrospective think aloud methods as techniques for assessing
online learning is limited (van den Haak, de Jong, & Schellens, 2003). Moreover, several
studies show that both the concurrent think aloud and retrospective report have weaknesses—the
concurrent think aloud has the disadvantage of completing two tasks simultaneously while the
retrospective think aloud has the disadvantage of thinking aloud after the task has been
completed (Branch, 2006; Kuusela & Paul, 2000; McDonald, Edwards, & Zhao, 2012; van Gog,
Kester, Nievelstein, Giesbers, & Paas, 2009). Findings of such studies suggest that alternative
think aloud methods warrant attention.
To address these gaps in the literature, study 1 compares three think aloud methods—
concurrent, retrospective, and virtual revisit—for examining elementary teachers’ cognitive
4
processes as they use a multimedia professional development website. The virtual revisit is a
variation of the retrospective think aloud method and allows participants to review and comment
on a visual recording of how they interacted with a particular website. The goal of the virtual
revisit is to aid recall of original events and thought processes by using a screen-capture
recording of participants’ navigational experiences. Similar to cued retrospective reporting
where participants are given instructions to report retrospectively on the basis of a record of
observations (van Gog, Paas, van Marrienboer, & Witte, 2005), the virtual revisit think aloud
combines a retrospective think aloud with screen capture technology to aid recall of original
events and thought processes.
Study 1 involves 45 elementary teachers from the Greater Toronto Area. Participants
were assigned to one of three think aloud conditions—concurrent, retrospective, and virtual
revisit. They were all asked to use the website as they would normally do when seeking
information related to their teaching practices but the three groups were asked to think aloud
about their site navigation under different conditions. Participants in the concurrent condition
verbalized their thoughts while simultaneously navigating the website for 20 minutes;
participants in the retrospective condition verbalized their thoughts immediately following their
20-minute navigation without any aids; and participants in the virtual revisit condition
verbalized their thoughts while viewing a screen recording of their 20-minute navigation.
Comparisons were made between concurrent, retrospective, and virtual revisit think aloud
procedures. Think aloud protocols, screen recordings, and interview data were analyzed using
quantitative and qualitative methods. Study 1 results show differences in the types and
complexity of verbalizations produced by participants in each condition (p < .05). Furthermore,
results reveal differences in website task completion and participants’ reported comfort levels
with thinking aloud. Overall findings of study 1 show benefits and limitations to each type of
5
think aloud method in the context of online teacher professional development. Findings also
suggest that the virtual revisit think aloud method provides a comprehensive picture of
elementary teachers’ cognitive processes as they use a professional development website.
Following findings from study 1, study 2 examines data from the virtual revisit condition
(n = 15). The findings of study 1 showed that participants assigned to the virtual revisit think
aloud produced rich and thorough verbalizations related to higher cognitive processes (e.g.,
planning, reasoning, reflecting). Therefore, a second study was conducted to analyze the think
aloud transcripts, interview data, and questionnaire data from the virtual revisit condition.
While research that examines teacher professional development is extensive and includes
multiple contexts—course instruction (Podhajski, Mather, Nathan, & Sammons, 2009), in-
school professional communities (Porche, Pallante, & Snow 2012), and web-based learning
environments (Downer et al., 2012)—research that examines the learning processes and
experiences of elementary teachers as they use informal professional development websites is
limited (Bakkenes, Vermunt, & Wubbels, 2010; Downer et al., 2012; Hoekstra, Brekelmans,
Beijaard, & Korthagen, 2009; Vermunt & Endedijk, 2011). Therefore, study 2 utilizes the
virtual revisit think aloud method and two other data sources (interview and questionnaire data)
to examine elementary teachers’ learning experiences as they navigate the given website. This
topic warrants attention for two main reasons: elementary teachers are increasingly using the
Internet for their professional learning, and “to understand teacher learning we must study it
within multiple contexts” (Borko, 2004, p. 4).
Study 2 utilizes qualitative methods to analyze the think aloud transcripts, interview
data, and questionnaire data from the virtual revisit condition. Specifically, study 2 follows a
grounded theory approach to gain an in-depth understanding of 15 participants’ learning
experiences as they navigate The Balanced Literacy Diet website. Three phases of coding—
6
open coding, axial coding and selective coding—led to a theoretical model describing: (1)
conditions that affect how elementary teachers use and learn from a professional development
website, (2) the central phenomenon, navigating a professional development website, as
influenced by the conditions, (3) navigational strategies, and (4) potential learning outcomes.
Thesis Overview
The remainder of this thesis includes four chapters. Chapter 2 provides the general
methodology relevant to both study 1 and study 2. The context, participant recruitment,
demographic characteristics, data sources, and procedure are similar for study 1 and study 2;
therefore, to reduce redundancy as much as possible, the second chapter presents the general
methodology for this thesis.
Chapter 3 presents study 1, an empirical study that compares the relative utility of three
think aloud methods: concurrent, retrospective, and virtual revisit. First, Chapter 3 discusses
two main topics relevant to study 1: (1) the think aloud methodology, and (2) teacher cognition
and cognitive processes. Chapter 3 continues with a description of the study purpose and
research questions, the relevant methods, and the results and discussion. Chapter 3 concludes
with a general discussion, including study 1 limitations, significance and implications, and
future directions.
Based on the findings of study 1, Chapter 4 presents study 2, an in-depth qualitative
study that examines the online learning experiences of participants from the virtual revisit think
aloud condition. First, Chapter 4 describes three theoretical models for understanding teacher
learning: adult learning theory, self-directed learning, and Shulman and Shulman’s theory of
teacher learning (2004). Components of each learning theory are discussed with respect to
online teacher professional development. Chapter 4 continues with a description of the study
purpose and research questions for study 2, the relevant methods for study 2, and the findings
7
and discussion. Chapter 4 concludes with a general discussion, including study limitations,
significance and implications, and future directions.
Chapter 5, the final chapter of the thesis, presents a discussion of the general conclusions
derived from the two studies. The final chapter also reflects on the research and educational
implications of the thesis as a whole.
8
CHAPTER 2
General Methodology
This chapter describes the general methodology for study 1 and study 2. As previously
mentioned, the context, participant demographic characteristics, data sources, and procedure are
similar for both studies; thus the purpose of including a general methodology chapter is to
describe the common components of the study 1 and study 2 method sections. Chapter 3 and
chapter 4 will describe details of the methods pertaining to each study, including the research
design and data analysis. This chapter begins with a description of the context of the current
dissertation, The Balanced Literacy Diet website, and highlights the website’s main features and
various web pages. Then, the chapter describes participant recruitment. The next section
presents a description of the four data sources: demographic questionnaire, think aloud, screen
capture technology, and semi-structured interview. This chapter ends with an outline of the
procedure.
Context
This dissertation research was undertaken within the context of the development and
refinement of a professional development website, The Balanced Literacy Diet: Putting
research into practice in the classroom (www.LitDiet.org). This website is a multimedia
evidence-informed literacy website that provides free professional learning resources such as
virtual tours of exemplary classrooms (PreK-6), video clips of expert teachers explaining and
demonstrating effective educational practices, detailed lesson plans, photos of teaching
materials, and exemplars of student work. In addition, The Balanced Literacy Diet website uses
a healthy diet framework as a metaphor for the various components of literacy at different stages
of development. These literacy components are consistent with The National Reading Panel’s
(2000) report on evidence-based literacy instruction. The report concluded that high quality
effective components of literacy instruction include phonological awareness, phonics, fluency,
9
vocabulary, and comprehension strategies. Reading researchers generally agree on these
components of literacy practices (Cunningham, Zibulsky, Stanovich, & Stanovich, 2009) and
several studies have shown a positive relationship between these components and student
growth in reading and writing (Downer et al., 2011; Porche et al., 2012; Pressley et al., 2007).
Taken together, The Balanced Literacy Diet website is content rich, interactive, and research-
informed; thus it offers an ideal context in which to examine elementary teachers’ use of
professional development websites while, at the same time, generating data which can serve to
further develop the effectiveness of the site as an online professional resource.
Figure 2.1 shows the homepage of The Balanced Literacy Diet website. The homepage
provides visitors of the site with five tabs from which to choose at the top of the page:
Homepage, Food Groups, Recipe Finder, Virtual Tours, and How To Videos. The homepage
also provides visitors with an overview of the site, sample virtual tours and lesson plans, and
relevant external links.
Figure 2.1
Homepage of The Balanced Literacy Diet website
10
The Food Groups tab brings site users to an introductory page on the different
components of literacy that contribute to a balanced literacy program. Site visitors can access
each literacy component, (e.g., phonemic awareness, reading comprehension, text structures and
genres) from the drop down Food Groups menu. The food group pages provide a combination
of theoretical and practical information on each component of literacy. The site visitor can
access this information visually, auditorily and textually.
From the Recipe Finder tab, site visitors can choose lessons and activities that range
from pre-kindergarten through sixth grade. Following the healthy diet theme, the lessons and
activities are referred to as recipes. Recipes include photos of instructional material and student
work, videos of the teacher explaining or demonstrating the lesson, and the activity objective
and implementation. For further reading, site visitors can also access additional resources from
the recipes, including journal articles and teacher guide books.
Site visitors can also visit classrooms virtually, through the Virtual Tour tab. Within
each virtual tour, users can navigate around a virtual classroom—they can zoom in to view
close-ups of teacher material and student work, and click on dots or “hot spots” to hear from the
classroom teacher about a specific lesson or area of the classroom.
The How To Videos tab brings users to the website’s YouTube channel. The Balanced
Literacy Diet YouTube channel provides over 500 videos of teachers explaining and
demonstrating literacy activities.
In addition, site visitors can access content pages relevant to their role (i.e., teachers,
administrators, parents, and teacher educators). These content pages include pedagogical
information and relevant external links. Appendix A-1 contains sample screen shots of each
type of webpage.
11
Participant Recruitment
Forty-five Toronto area practicing elementary teachers participated in this research over
an eight-month period. Participants were recruited through visits to Additional Qualifications
(AQ) courses, recruitment posters, and by word of mouth. First, elementary teachers were
recruited through AQ courses at the Ontario Institute for Studies in Education (OISE). A
request to contact AQ course instructors was sent in an email to the OISE AQ office. The email
described the study purpose and participant criteria (e.g., grades taught ranging from
kindergarten to grade six, practicing teachers). The AQ office replied with a contact list of 15
AQ course instructors. Course instructors were contacted by email and asked for permission to
attend one of their classes to invite teachers to participate in the research. Appendix A-2
presents the invitation to participate script. Ten instructors replied to the email. Seven agreed to
a course visit and three agreed to email the information to the teachers in their course. The
course visits involved a verbal explanation of the research purpose and participant involvement,
and a request for teachers to take part in the study. Interested teachers were given an envelope
with an information letter and consent form explaining the research in greater detail. Appendix
A-3 contains the information letter and consent form. Interested teachers were asked to include
their email address on the consent form so that the questionnaire link could be emailed to them.
Envelopes were collected at the time of the visit and at the end of the class. Instructors who
were willing to forward the information to the teachers enrolled in their course were sent the
explanation of the research purpose and the description of participant involvement. The total
number of teachers contacted through AQ courses was approximately 200 (with approximately
20 teachers per class). Many teachers initially showed interest in participating during the course
visits. However, given the criteria for participation and the reality that practicing teachers are
12
very busy, the response rate was low; only four teachers became study participants despite a $20
gift card incentive.
Due to the difficulty recruiting practicing primary and junior level teachers through AQ
course visits, recruitment also occurred through study posters and by word of mouth. Study
posters were displayed at OISE and sent via email by colleagues interested in the research.
Appendix A-4 presents the recruitment poster. Potential participants were also contacted by
teachers who completed the study. Of the 45 teachers who agreed to participate 28 were
recruited through the recruitment posters, four were recruited from an AQ course, and 13
teachers were recruited by word of mouth. Demographic characteristics are described below.
Due to the challenges of participant recruitment, a non-random sample was used in this
research. A distribution chart shows the demographic characteristics across the three different
modes of recruitment (see Appendix A-5).
Data Sources
This research uses four different data sources: a demographic questionnaire, think aloud,
screen capture technology, and a semi-structured interview.
Demographic Questionnaire
A demographic questionnaire (see Appendix A-6) was administered to participants to
obtain data on a range of relevant factors based on the literature related to teacher development
and online learning (e.g., age, gender, years of teaching experience, education, extent of
involvement with various web-based technologies). Participants were asked to complete the
questionnaire online prior to a one-on-one meeting. The questionnaire was administered
through Survey Planet (https://www.surveyplanet.com/), an online survey program. After
participants completed the questionnaire, they were placed in one of three think aloud conditions
(concurrent, retrospective, and virtual revisit). The conditions are described below. Stratified
13
random assignment was employed to ensure that certain demographic features were represented
within each group as equally as possible. Stratification variables were selected based on the
literature on web navigation. Research has found that gender (Page, Robson, & Uncles, 2012; J.
Pearson, A. Pearson, & Green, 2007), age (Laberge & Scialfa, 2005), subject matter knowledge
and experience (Laberge & Scialfa, 2005), and computer self-efficacy (Page et al., 2012)
influence the perception and navigation of websites; therefore, comparable aspects including
gender, age, years of teaching experience, current teaching grade, comfort with technology, and
frequency of Internet use for professional purposes, were selected as the key variables used for
the group assignment. A one-way ANOVA was conducted to compare the three conditions to
determine if the conditions had any significant differences with respect to these key variables.
The conditions did not differ significantly on any of these key variables.
The results of the demographic questionnaire revealed that, in the sample overall, male
participants (n = 9, 20%) represented a smaller proportion of the sample than female participants
(n = 36, 80%). This finding is consistent with the proportion of few male elementary teachers
in primary and junior positions (Jamieson, 2007). The participants represented a range of age
and years of experience with the majority of teachers being 30-34 years of age (n = 34, 27%)
and most teachers having 5-9 years of experience (n = 17, 38%). Grades taught ranged from
kindergarten through sixth grade. Two teachers (4%) taught kindergarten, 17 teachers (38%)
taught at the primary level, 19 teachers (42%) taught at the junior level, and seven teachers
(16%) taught across divisions, at the primary and junior levels.
Results from the demographic questionnaire revealed similarities across participants in
terms of comfort using the Internet and frequency of Internet use for professional purposes. The
majority of teachers reported feeling very comfortable using the Internet (n = 41, 91%).
14
Furthermore, the majority of teachers reported using the Internet for professional purposes at
least once a day (n = 21, 47%).
Table 2.1 summarizes the demographic characteristics for the participants across the
conditions. As can be seen from the table, participants in the three groups were quite similar
with respect to gender, age, years of teaching, current grade, comfort with technology, and
frequency of Internet use. In other words, the stratified random assignment was successful. As
mentioned above, statistical tests found no significant differences between groups on these
variables.
Table 2.1
Demographic Characteristics
Characteristic Concurrent
Condition
(n = 15)
Retrospective
Condition
(n = 15)
Virtual
Revisit
Condition
(n = 15)
Total
(N = 45)
n (%) n (%) n (%) n (%)
Gender
Male 3 (20%) 3 (20%) 3 (20%) 9 (20%)
Female 12 (80%) 12 (80%) 12 (80%) 36 (80%)
Age
25-29 3 (20%) 3 (20%) 3 (20%) 9 (20%)
30-34 3 (20%) 5 (33%) 4 (27%) 12 (27%)
35-39 4 (27%) 3 (20%) 4 (27%) 11 (24%)
40-44 2 (13%) 2 (13%) 2 (13%) 6 (13%)
45-49 1 (6%) 1 (6%) 1 (6%) 3 (7%)
50-54 2 (13%) 0 0 2 (4%)
55+ 0 1 (6%) 1 (6%) 2 (4%)
Years
Teaching
2-4 years 3 (20%) 3 (20%) 4 (27%) 10 (22%)
5-9 years 6 (40%) 6 (40%) 5 (33%) 17 (38%)
10-14 years 3 (20%) 5 (33%) 4 (27%) 12 (27%)
15-19 years 1 (6%) 0 1 (6%) 2 (4%)
20-24 years 1 (6%) 0 1 (6%) 2 (4%)
25+ years 1 (6%) 1 (6%) 0 2 (4%)
Current
Grade
Kindergarten 1 (6%) 1 (6%) 0 2 (4%)
Primary 5 (33%) 5 (33%) 7 (47%) 17 (38%)
Junior 7 (47%) 6 (40%) 6 (40%) 19 (42%)
15
Primary/Junior 2 (13%) 3 (20%) 2 (13%) 7 (16%)
Comfort
Using
Internet
Very 15 (100%) 13 (87%) 13 (87%) 41 (91%)
Somewhat 0 2 (13%) 2 (13%) 4 (9%)
Not very 0 0 0 0
Frequency of
Internet Use
for
Professional
Purposes
> once/day 5 (33%) 3 (20%) 6 (40%) 14 (31%)
Once/day 6 (40%) 8 (53%) 7 (47%) 21 (47%)
Once/week 4 (27%) 4 (27%) 1 (6%) 9 (20%)
Once/month 0 0 1 (6%) 1 (2%)
Think Aloud
Participants completed a think aloud during a one-on-one meeting with the primary
researcher. A digital recorder was used to audio record participants’ verbalizations.
Participants were asked to complete a second consent form giving permission to record the think
aloud and interview (see Appendix A-7).
Concurrent think aloud. Participants in the concurrent condition verbalized their
thoughts for 20 minutes while simultaneously completing a website task.
Retrospective think aloud. Immediately following a 20-minute website task,
participants in the retrospective condition recalled and verbalized their thought processes
without any aids.
Virtual revisit think aloud. Immediately following a 20-minute website task,
participants in the virtual revisit condition reviewed their online choices virtually and verbalized
their thoughts while viewing the 20-minute screen recording of their explorations.
16
Screen Capture Technology
During participants’ navigation of The Balanced Literacy Diet website, each visual step
was captured with Camtasia Studio, a screen-recording computer software program developed
by TechSmith (Uppal, 2011). Appendix A-8 displays a screen shot of Camtasia Studio that
includes an image of a screen recording of one participant’s navigation. The purpose of using
screen capture technology was to gain greater insight into participants’ web-based actions and
behaviours. Web tracking through screen-recordings offers a deeper understanding of
information-seeking phenomena on the Internet (Fourie & Bothma, 2007).
Semi-structured Interview
A semi-structured interview (see Appendix A-9) followed the participants’ website
navigations. A digital recorder was used to record the interviews. Questions concerned
participants’ general feelings about their navigational experiences as well as their comfort levels
while thinking aloud, whether verbalizing their thoughts concurrently, retrospectively or via the
virtual revisit.
Procedure
One-on-One Meeting
Upon completion of the questionnaire, participants met with the primary research for a
one-on-one meeting. Meetings occurred at a time and location that was most convenient to the
participants (e.g., participant’s school or home, OISE). An open location led to an increase in
teacher interest and willingness to participate in the study. The one-on-one meetings followed a
sequence of events and lasted approximately 45 minutes. First, the primary research restated the
purpose of the research. Next, participants completed the website task and think aloud. Finally,
the semi-structured interview was conducted.
17
Website task and think aloud. The following website task instructions were presented
to all participants:
Your task is to use The Balanced Literacy Diet website as you normally would when
seeking information online for your teaching practices.
While the website task instructions were consistent across the conditions, the think aloud
instructions varied for each condition. Participants in the concurrent condition were given the
think aloud instructions before they completed the website task, whereas participants in the
retrospective and virtual revisit groups were given the think aloud instructions after they
completed the website task. The purpose of informing participants in the retrospective and
virtual revisit conditions of the think aloud instructions after their navigation was to reduce
reactivity—“influences of the verbalizations on the decision process” (Ranyard & Svenson,
2010, p. 119)—as much as possible. The following passage outlines the think aloud
instructions. The underlined portions state the different think aloud instructions given for each
condition.
In this study we are interested in what you think about when you explore a professional
development website. In order to do this I am going to ask you to think aloud
(concurrent condition: as you explore The Balanced Literacy Diet website; retrospective
condition: about your exploration of The Balanced Literacy Diet website; virtual revisit
condition: while you view a recording of your exploration of The Balanced Literacy Diet
website). What I mean by think aloud is that I want you to tell me everything that you
are/were thinking from the time you begin/began exploring the website until the end of
your exploration. I would like you to talk aloud constantly. I don’t want you to try to
plan out what you say or try to explain to me what you are saying. Just act as if you are
alone in the room speaking to yourself. It is most important that you keep talking.
18
While think aloud studies most often employ a specific task, an open-ended task was
used in this study to reflect as naturally as possible how teachers use and learn from professional
development websites. To reduce disruption to the participants’ cognitive processes, prompts,
redirections, and interventions were kept to a minimum during the process of verbalizing
(Jaspers, 2009). Participants who were silent for a period of 30 seconds were only told to “keep
talking.” This prompt was given to one participant in the concurrent condition.
Prior to their navigation of The Balanced Literacy Diet website, participants in the
concurrent condition were given a short warm-up exercise in thinking aloud. This exercise was
completed on another educational website, Reading Rockets (http://www.readingrockets.org/),
and lasted approximately 30 seconds to 1 minute. The warm-up exercise included instructions
that were consistent with the concurrent think aloud procedure. The practice task provided the
participants in the concurrent condition with an opportunity to become familiar with thinking
aloud while simultaneously navigating the website. Feedback was provided to ensure that the
participants continuously verbalized their thoughts while navigating the website (van Someren
et al., 1994). Participants in the retrospective and virtual revisit conditions were not given a
warm up task for two reasons: (1) to reduce reactivity as much as possible and, (2) to reduce the
time lapse between participants’ navigation and their think aloud. Feedback was provided if
participants in either of these two conditions requested more information or clarity.
Semi-structured interview. The semi-structured interview was conducted following the
completion of participants’ website task and think aloud. At the end of the interview,
participants were thanked for participating in this research and given a $20 gift card for their
contribution.
19
CHAPTER 3
Study 1: Comparing Three Think Aloud Methods for Examining Teachers’ Cognitive
Processes as They Use a Multimedia Professional Development Website
Chapter 3 presents a complete empirical study in seven major sections: (1) a literature
review, (2) the study purpose and research questions, (3) methods, (4) results and discussion, (5)
general discussion and limitations, (6) significance and implications, and (7) conclusion. First,
the literature review discusses the think aloud methodology, teacher cognition, and cognitive
processes. Next, this chapter presents the study purpose and research questions. The third
major section of this chapter is the methods section, which begins with an overview of the
research design. The methods section continues with the data analysis and highlights three
types of analyses: the think aloud analysis, time sampling analysis, and interview analysis. The
methods section concludes with a description of the data review process. The fourth major
section of this chapter is the results and discussion. A general discussion including the
limitations of study 1 follows the results and discussion section. Finally, this chapter ends with
the study significance, research implications, and a conclusion.
The following review of the literature includes two main topics: think aloud
methodology, and teacher cognition. The purpose of including a literature review in these areas
is twofold: (1) this study compares three think aloud methodologies, and (2) this study examines
teachers’ cognitive processes as they use a professional development website; therefore, the
following review sets a foundation for study 1.
Literature Review
Think Aloud Methodology
Thinking aloud has historical roots in introspection analysis, a form of data collection
aimed at investigating psychological claims and theories of mind during the eighteenth century
(Boren & Ramey, 2000; Ericsson, 2002). The cognitive revolution of the 50s and 60s produced
20
alternative types of verbal reports of thinking to gather information about cognitive structures
and processes (Ericsson, 2003). Today, the think aloud method most widely employed is based
on the techniques of protocol analysis by Ericsson and Simon (1984, 1993). As described by
Ericsson and Simon (1984), thinking aloud captures cognitive processes in real time and verbal
reports “provide the most informative data available on thinking during cognitive tasks”
(Ericsson, 2003). Cognitive processes underlying decisions and behaviours are usually “hidden
from direct observation” (Gaissmaier, Fifc, & Rieskany, 2010, p. 141). However, the think
aloud method makes monitoring cognitive processes possible—the think aloud generates direct
data about the ongoing cognitive processes that occur during task performance (Jaspers, Steen,
van den Bos, & Geenen, 2004).
Ericsson and Simon (1984) describe three levels of verbalizations that can occur during
the think aloud method. The first two levels require information processing in the participant’s
short term memory and the third level requires additional cognitive resources and retrieval of
information from long term memory (Olmsted-Hawala, Murphy, & Hawala, 2010). While
Ericsson and Simon (1984) state that Level 3 verbalizations or higher cognitive processes are
less reliable because they involve access to long-term memory, usability researchers suggest that
this type of data provides useful information about online learning, website user goals, and
online behaviours (Boren & Ramey, 2000; Guan, Lee, Cuddihy, & Ramey, 2006; Olmsted-
Hawala et al., 2010).
Usability researchers most often employ the concurrent and retrospective think aloud
methods to gain insight into web seeking behaviours and to evaluate a website’s content and
ease of use (Aranyi, Schaik, & Barker, 2012; Barzilai & Zohar, 2012; Branch, 2006; Kuusela &
Paul, 2000). As previously mentioned, during the concurrent procedure participants verbalize
their thoughts aloud while they simultaneously complete a task. Verbal reports that result from
21
the concurrent procedure generate data about the website user’s navigational experience. For
instance, Aranyi and his colleagues (2012) conducted an exploratory study of interaction
experience with a news website. The concurrent think aloud yielded five categories of
experience based on the participants’ evaluative statements: impression, content, layout,
information, architecture, and diversion (Aranyi et al., 2012). Similarly, Barzilai and Zohar
(2012) utilized the concurrent procedure to examine epistemic thinking in action. Data was
collected to shed light on the relationship between sixth grade students’ knowledge construction
and their online practices. Analysis revealed a positive relation between students’ online
strategies and their epistemic cognition (Barzilai & Zohar, 2012). Damico and Balidon (2007)
also employed the concurrent procedure to examine how elementary students engage with an
educational website. Findings from their study highlight how elementary students evaluated
claims and evidence of online educational resources (Damico & Balidon, 2007).
The retrospective procedure is also referred to as post-task testing, retrospective report,
and think after. Retrospective think alouds alone are used less often in the fields of online
learning and website usability since they require participants to think aloud after a task has been
completed. An international survey found that just 5% of think aloud studies (not limited to
website usability) employed the retrospective technique, whereas 89% used the concurrent think
aloud, and 6% used an alternative think aloud (McDonald et al., 2012).
One of the main reasons why retrospective think alouds are used less often is due to the
fact that the procedure relies on the ability to recall decisions after a task has been completed.
As participants recall their decisions, information may be incomplete and include errors,
omissions, and substitutions (Branch, 2006). For instance, a comparative study that examined
retrospective and concurrent verbal protocol analysis in the context of a decision-making task
22
found retrospective reports more prone to errors of omission whereas concurrent reports
contained more relevant information about the decision making process (Kuusela & Paul, 2000).
While retrospective procedures are limited by the fact that they may be incomplete and
include errors, omissions, and substitutions (Branch, 2006), they have the advantage of freeing
cognitive resources by thinking aloud after the task has been completed—retrospective think
alouds do not interfere with task performance (McDonald et al., 2012). Concurrent think
alouds, on the other hand, can interfere with task performance since participants verbalize their
thoughts while they simultaneously complete a given task—participants engage in two different
processes at the same time. When two processes occur simultaneously there is an increase in
cognitive load—“the level of mental energy required to process a given amount of information”
(Ping Lim, 2004, p. 17). As a result of a higher cognitive load, task completion may be
compromised during the concurrent procedure and resulting think aloud reports are often
procedural in nature (McDonald et al., 2012; van Gog, Kester, Nievelstein, Giesbers, & Paas,
2009).
Despite the limitations of the concurrent procedure, it has been widely used in usability
research, mostly as a means to evaluate a given website—participants verbalize their thoughts
about the ease of use and accessibility of information. While evaluative data contributes to the
refinement of websites, user experience is a complex process and usability research should go
beyond evaluating websites to include a range of cognitive processes and learning strategies
(Dillon, 2001). Moreover, few studies have been undertaken to compare the relative utility of
different think aloud procedures; and no research, to the investigator’s awareness, has compared
the relative utility of the concurrent, retrospective, and virtual revisit think aloud procedures
with practicing elementary teacher participants. Taken together, the current study addresses
these gaps in the literature by examining the utility of three think aloud methods for
23
understanding elementary teachers’ cognitive processes as they use a multimedia professional
development website. Given that this study examines elementary teachers’ cognitive processes,
the next section provides an overview of teacher cognition and cognitive processes.
Teacher Cognition
Teacher cognition can be described simply as how teachers process information. In
greater complexity, teacher cognition integrates theoretical and practical knowledge and is
guided by a set of organized beliefs, professional goals, and expectations (Bakkenes et al., 2010;
Borg, 2006). Teacher cognition research has examined several areas, including teacher
planning, differences between expert and novice teachers, and teacher decision-making during
instruction (Borko, Livingston, & Shavelson, 1990; Cercone, 2008; Krathwohl, 2002; Mercier,
Girard, Brodeur, & Laplante, 2010). With respect to teacher planning, research has identified
the foci of teachers as they plan for specialty topics and core subjects (Borko et al., 1990; Wing-
mui So, 1997). Research that reports on how teachers plan has also described factors that
influence the planning process: for example, information about students, the nature of the
instructional task, and the context of instruction (Borko et al., 1990; Warren, 2000).
Research on teacher cognition has also identified differences between expert and novice
teachers during lesson planning and implementation (Westerman, 1991). For instance, one
major difference is how expert and novice teachers perceive their students: expert teachers
usually consider individual learners during planning, whereas novice teachers are more likely to
consider the class as a whole during planning (Hogan, Rabinowitz, & Craven, 2003). In
addition, distinct differences in pedagogical knowledge, pedagogical content knowledge,
instructional approaches, and communication have been found to exist between expert and
novice teachers (Hogan et al., 2003; Livingston & Borko, 1989).
24
Teacher cognition research has also identified underlying reasons for instructional
decisions (Rich & Hannafin, 2008). For instance, in a study that examined preservice teacher
decision-making, participants identified several teaching strategies they employed during
instruction (e.g., asking questions, guiding, and modeling). These decisions were accompanied
by a specific reason. For example, pedagogical reasons (underlying beliefs about the nature of
learning) were reported as reasons for employing modeling as an instructional strategy (Rich &
Hannafin, 2008).
Cognitive processes underlie the study of teacher decision-making during their
professional learning experiences. Cognitive processes can be categorized as higher order
processes and lower order processes. Higher order processes, such as reasoning, involve access
to “thematically related information in long-term memory” (Horz & Schnotz, 2010, p. 238).
Information stored in memory “is interrelated and rearranged, and extended to achieve a specific
purpose” (Lewis & Smith, 1993, p. 136). Lower order processes, such as procedural
knowledge, are normally executed in an automated way and are “only marginally influenced by
intentional processes” (Horz & Schnotz, 2010, p. 238). Lower order processes provide a
foundation for higher levels of thinking. In the current study, higher order cognitive processes
refer to planning, connecting, reasoning, reflecting, and evaluating, whereas lower order
cognitive processes refer to diversion, understanding, describing, and reading. These cognitive
processes are described in detail in the methods section of this chapter.
As previously described, online learning is a favored approach to teacher professional
development. Given that elementary teachers are increasingly accessing professional
development websites, it is essential to examine teachers’ cognitive processes as they use and
learn from these online environments. Moreover, methods for understanding how teachers think
and learn within online environments need to be examined and compared so that researchers can
25
gain a comprehensive picture of teachers’ cognitive processes and learning experiences within
online learning environments.
Research Questions
This study is exploratory in nature and is not aimed at verifying hypotheses. In addition,
this study is context-specific and uses one professional development website. The following
research questions guided the study:
1. To what extent do participants’ verbalizations differ across the three think aloud
methods?
2. How do think aloud tasks differ across the three think aloud conditions?
3. To what extent does comfort level with thinking aloud differ across the three think
aloud methods?
Methods
The methods section of this chapter begins with a description of the research design and
overall process for study 1. Next, the data analysis highlights three analyses and their
procedures: the think aloud analysis, time sampling observation analysis, and interview analysis.
The methods section of study 1 concludes with a description of the data review process—a
process of inspecting the data for normality and equal variance prior to performing any
statistical tests.
Research Design
Study 1 employs a convergence mixed methods research design. The purpose of the
convergence mixed methods design is to “end up with valid and well-substantiated conclusions
about a single phenomenon” (Creswell & Piano Clark, 2007, p. 65). This study utilizes the
approach to combine statistical results with qualitative findings to better understand the relative
usefulness of three think aloud procedures for examining elementary teachers’ cognitive
26
processes as they explore a multimedia professional development website. Furthermore, the
qualitative analyses are used to corroborate and expand upon the quantitative results,
specifically in relation to research question two: How do think aloud tasks differ across the three
think aloud conditions?; and research question three: To what extent does comfort level with
thinking aloud differ across the three think aloud methods?
The research process for study 1 begins with four different data sources: demographic
questionnaire, think aloud, screen capture technology, and semi-structured interview. The main
purpose of the demographic questionnaire for study 1 was to ensure that certain demographic
features were represented within each group as equally as possible; therefore, the data set for
study 1 includes only three components: think aloud transcripts, 20-minute screen recordings,
and interview transcripts. These components were analyzed separately in an attempt to answer
the three research questions as follows:
(1) Question 1: To what extent do participants’ verbalizations differ across the three
think aloud methods?—Descriptive statistics, analysis of variance (ANOVA), and
Pearson correlations were performed on the think aloud transcripts. These
procedures are described in the data analysis section under the heading, Think Aloud
Analysis.
(2) Question 2: How do think aloud tasks differ across the three think aloud
conditions?—Descriptive statistics, ANOVA, Pearson correlations, and qualitative
observations were performed on the screen recording data. These procedures are
described in the data analysis section under the heading, Time Sampling Observation
Analysis.
(3) Question 3: To what extent does comfort level with thinking aloud differ across the
three think aloud methods?—Descriptive statistics, ANOVA, Pearson correlations,
27
and a general inductive approach to analysis were performed on portions of the
interview transcripts. These procedures are described in the data analysis section
under the heading, Interview Analysis.
Data Collection
Date sources:
demographic questionnaire
think aloud
screen capture technology
semi-structured interview
Study One Data Set
Three components:
45 think aloud transcripts
45 screen recordings
45 interview transcripts
Research Question 1: To what extent do
participants’ verbalizations
differ across the three think
aloud methods?
Think aloud analysis:
1. Descriptive statistics,
ANOVA, and Pearson
correlations were
performed on think aloud
transcripts
Research Question 2:
How do think aloud tasks
differ across the three think
aloud conditions?
Time sampling
observation analysis: 1. Descriptive statistics,
ANOVA, and Pearson
correlations were
performed on screen
recording data
2. Qualitative observations
were performed on
screen recordings
Research Question 3:
To what extent does
comfort level with
thinking aloud differ
across the three think
aloud methods?
Interview Analysis
1. Descriptive statistics,
ANOVA, and Pearson
correlations were
performed on
interview transcripts
2. A qualitative approach
to analysis was
performed on
interview transcripts
Interpretative Phase
-Results merged to produce a more
complete understanding of the single
phenomenon
Figure 3.1. Convergence mixed methods research design
28
Results of the analyses converged during the interpretive phase—a phase of analysis that
produces a more complete understanding of the main phenomenon (Creswell & Piano Clark,
2007). This chapter presents the interpretative phase in the general discussion section. Figure
3.1 outlines the overall process for study 1. The following data analysis section describes the
three different analyses and their procedures.
Data Analysis
This section presents the analysis procedures. First, the think aloud analysis describes
the procedures performed on the think aloud transcripts. Secondly, the time sampling
observation analysis describes two procedures performed on the 20-minute screen recordings.
Finally, the interview analysis describes two procedures performed on portions of the interview
transcripts.
Think aloud analysis. The think aloud analysis involves six events: (1) transcription
and word counts, (2) the development of a coding scheme, (3) unitizing the transcripts, (4) inter-
rater agreement, (5) coding the think aloud transcripts, and (6) performing statistical tests.
1. Transcription and word counts. Audio recordings were transcribed verbatim
resulting in 45 think aloud transcripts. The primary researcher transcribed all 45 recordings to
ensure greater transcription quality—errors, such as omissions and substitutions, are less likely
to occur when the transcriber is familiar with the research (Poland, 1995; Tilley, 2003).
Word counts were calculated for each of the three conditions (concurrent, retrospective,
and virtual revisit). As shown in Table 3.1, there are clear differences in the average number of
words participants generated in the three think aloud conditions. These word counts were
analyzed and results from this analysis are presented in the results and discussion section. On
average, transcripts that resulted from the concurrent think aloud contained 1676.13 words (SD
= 394.89), transcripts that resulted from the retrospective think aloud contained 658.80 words
29
(SD = 229.00), and transcripts that resulted from the virtual revisit think aloud contained
2637.87 words (SD = 359.87). Table 3.1 presents the average number of words across the three
conditions.
Table 3.1
Average Word Counts across the Three Conditions
Concurrent
Condition
Retrospective
Condition
Virtual
Revisit
Condition
M
(SD)
M
(SD)
M
(SD)
Word Count
1676.13
(394.89)
658.80
(229.00)
2637.87
(359.94)
2. Development of a coding scheme. The coding scheme used to code all 45 transcripts
was developed through a series of steps. First, an initial coding scheme was generated based on
several studies relating to website usability (Aranyi et al., 2012; Cooke, 2010; Damico &
Balidon, 2007; van Gog et al., 2005; Tan & Wei, 2006; Zhao & McDonald, 2010), teacher
planning and decision making (Kansanen et al., 2000; Moos, 2014), and Ericsson and Simon’s
levels of verbalizations and suggested statements (1984). Fifty-three potential categories
comprised the initial list (see Appendix B-1). Categories were refined, interconnected, and
eliminated resulting in a coding scheme of nine categories. The following three examples show
how categories were refined, interconnected, and eliminated:
(1) the category, Identifying and making sense of new information, was refined to
Understanding;
(2) the three categories, Procedure: describing current or future activity, Observation:
making an observation about the website or participant’s own behaviour, and Cognitive
mapping: taking in sensory information were found to be similar and therefore grouped
together as Describing; and
30
(3) the category, Verbal fillers, was eliminated as a result of re-examining the research
questions in relation to the initial list (verbal fillers, such as “um”, were deemed
uninformative).
To test this coding scheme, two transcripts were coded from each condition. As a result,
two categories, Evaluating and Describing, were separated into sub-categories. Evaluating was
separated into two groups: (1) Evaluating website content, and (2) Evaluating user experience.
This decision resulted from a clear distinction in the participants’ evaluative statements.
Participants either evaluated the website content (i.e., “These anchor charts are really useful,
they’re very practical”) or their user experience (i.e., “It was nice also that you could sort by
grade and then re-filter by the food groups”). Describing was also separated into two groups:
(1) Describing procedural behaviour, and (2) Describing website content. This decision resulted
from a clear distinction in the participants’ descriptive statements. Participants either described
what they viewed on the screen (i.e., “It’s showing me words and how to sound them out and
how many sounds are in each one”) or they described their web-based actions (i.e., “I’m going
to go into the food groups section”). The final coding scheme used to code all 45 transcripts
includes 11 categories. Table 3.2 summarizes the coding scheme and offers a description of
each category and an example from the think aloud transcripts.
Table 3.2
Coding Scheme-Categories Used to Code the Transcripts
Verbalization
Category
Description Examples from the Transcripts
Planning Referring to program planning,
reorganizing information to form or
develop new ideas; constructing and
creating
“…that is a lesson I can just take and
tweak for my class right away.”
Connecting Recalling information; activating
prior knowledge in relation to
information presented on the site;
finding a past example or recalling a
concept
“This reminds me of the mini lessons that
I like to do at the beginning of the year.”
31
Reasoning Providing a rationale for making a
navigational decision; explaining why
“I was considering looking at text
structures and I ended up choosing
reading comprehension strategies because
I’m trying to make that one of the main
focuses of our reading program.”
Reflecting Making meta-comments in reference
to awareness of their own thinking
and learning style
“I like to go over everything first and
then go back and look at something more
in-depth.”
Evaluating-
Website
Content
Making judgments or expressing
opinions about an aspect of the
website or information presented on
the website
“This kind of photo tour really informs
me in terms of good practice for
classroom management and good
classroom environments.”
Evaluating-
User
Experience
Expressing a positive or negative
feeling towards the usability and
accessibility of the website and its
features
“It’s nice that the link is already there for
me, that I don’t have to type in a separate
search button or go onto Google, I can
just click onto the link.”
Diversion
Verbalizing difficulties, including
utterances where participants indicate
uncertainty and confusion
“Hmm, assessment, how do I go, so what
do I do? Enter? Search?”
“How do I go back? This is…am I doing
something wrong?”
Understanding
Identifying and making sense of new
information and web-based tools
“At this point I’m just looking at the
main page and I’m understanding how
the website is organized.”
Describing-
Procedural
Behaviour
Describing what they are doing or
going to do or just did; statements
about participants’ actions during
their navigation
“I’m just looking at the videos right
now.”
Describing-
Website
Features
Describing the spatial characteristics,
website features and images; what
participants notice
“It’s showing various pictures and
monthly virtual tours.”
Reading Reading words, phrases or sentences
off the screen; reading out texts,
headings, links, and other on-screen
text
“Motivation for literacy, oral language,
knowledge building, concepts of print,
writing conventions.”
The categories listed in Table 3.2 are types of cognitive processes. As previously
discussed, cognitive processes can be categorized into two levels: higher order processes and
32
lower order processes. The higher order and more complex cognitive processes “involve the
manipulation of information” (McLoughlin & Mynard, 2009, p. 148), whereas the lower order
cognitive processes demand only “mechanical application of previously acquired information”
(Lewis & Smith, 1993, p.133). As previously stated, this study categorizes higher order
cognitive processes as planning, connecting, reasoning, reflecting, and evaluating. These
cognitive processes go beyond what is written in the text and relate a given concept to the
learner’s experience. In addition, this study categorizes lower order cognitive processes as
diversion, understanding, describing, and reading. The execution of these lower order cognitive
processes occurs in a more automated way.
The organization of the categories into higher and lower order cognitive processes is
consistent with Krathwohl’s revised taxonomy of educational objectives (2002). Krathwohl
(2002) organizes six major categories and 19 sub-categories of the cognitive domain
hierarchically and discusses how they differ in complexity (see Appendix B-2 for Krathwohl’s
Revised Taxonomy). Krathwohl (2002) distinguishes between higher and lower order cognitive
processes: the more complex categories (e.g., create) are higher on the scale, whereas the less
complex categories (e.g., recalling) are lower on the scale. This distinction was taken into
consideration during the development of the coding scheme for the current study.
3. Unitizing. Prior to coding, the think aloud transcripts were first segmented or
“unitized” into thought units—each utterance was deemed a separate segment or thought unit if
it conveyed relevant information and was preceded and followed by a pause and a change of
ideas (Lincoln & Guba, 1985). Unitizing involves subjective interpretation and
contextualization (Lomard, Snyder-Duch, & Campanella Bracken, 2004). Therefore, the
transcripts were unitized by the primary researcher, who had a thorough understanding of the
research topic.
33
Once all 45 transcripts were unitized, the total thought units were calculated. Similar to
the differences in word counts, there are clear differences in the average number of thought units
generated by participants in each think aloud condition. As with the word counts, the thought
units were analyzed, and results are presented in the results and discussion section. On average,
transcripts that resulted from the concurrent think aloud contained 147.40 thought units (SD =
35.83), transcripts that resulted from the retrospective think aloud contained 30.73 thought units
(SD = 13.15), and transcripts that resulted from the virtual revisit think aloud contained 123.27
thought units (SD = 20.58). Table 3.3 presents the average number of thought units across the
three conditions.
Table 3.3
Average Number of Thought Units across the Three Conditions
Concurrent
Condition
Retrospective
Condition
Virtual
Revisit
Condition
M
(SD)
M
(SD)
M
(SD)
Number of Thought
Units
147.40
(35.83)
30.73
(13.15)
123.27
(20.58)
4. Inter-rater agreement. Once the transcripts were unitized, a second coder was trained
on the coding scheme to establish inter-rater agreement. Training involved a review of the
research study questions, a review of the definitions used to code the transcripts, and a
demonstration of the coding process. Ten percent of the transcripts were coded and
disagreements were discussed and resolved until an inter-rater agreement of 97% was reached.
Cohen’s Kappa was performed to determine consistency among raters and was found to be 0.98,
CI (0.978-0.996).
5. Coding the transcripts. Due to the high level of inter-rater agreement, all 45
transcripts were coded by the primary researcher. NVivo (QSR International, 2012), a
34
qualitative data analysis software program, was used to code the transcripts. NVivo was chosen
for this study because it aids in the organization and management of large data sets. Thought
units were tallied to provide frequency counts for each category. These frequency counts were
then transformed into percentages based on the total number of thought units across categories.
Coding resulted in a total of 4,521 thought units. Among these, 102 (2.3%) thought
units were coded as planning, 354 (7.8%) as connecting, 939 (20.8%) as reflecting, 453 (10.0%)
as reasoning, 363 (8.0%) as evaluating user experience, 580 (12.8%) as evaluating website
content, 215 (4.8%) as understanding, 195 (4.3%) as diversion, 495 (11.0%) as describing
procedural behaviour, 318 (7.0%) as describing website features, and 507 (11.2%) as reading.
Table 3.4 displays the distribution of the frequency of thought units across the three conditions.
Table 3.4
Frequency of Thought Units across Conditions
Category Concurrent
Condition
Retrospective
Condition
Virtual Revisit
Condition
Total
Total Total Total Total %
Planning
9 20 73 102 2.26
Connecting
127 57 170 354 7.83
Reasoning
47 47 359 453 10.02
Reflecting
354 121 464 939 20.77
Evaluating Website
Content
229 66 285 580 12.83
Evaluating User
Experience
166 59 138 363 8.03
Diversion
136 3 56 195 4.31
Understanding
141 14 60 215 4.76
Describing
Procedural
Behaviour
265 57 173 495 10.95
35
Describing Website
Features
235 17 66 318 7.03
Reading 502 0.00 5 507 11.21
6. Performing statistical tests. First, the frequencies of each thought unit were
converted to percentages for each participant (for example, a participant in the concurrent
condition verbalized 2 planning thought units out of a total of 228 thought units; therefore, 0.9%
of the total thought units from this participant were planning thought units). These percentages
were entered into SPSS for analysis. The percentage of thought units in each condition were
then analyzed quantitatively. Analyzing the percentage of thought units, as opposed to the
frequency of thought units, produces a more accurate representation of the cognitive processes
(Rosenzweig, Krawec, & Montague, 2011). This allows for a more accurate comparison of
thought units across conditions. The word counts and total number of thought units were also
analyzed quantitatively. Descriptive statistics, ANOVA, and Pearson correlations were
performed on the proportion of thought units, total word counts, and total number of thought
units across the three conditions (concurrent, retrospective, and virtual revisit). The results and
discussion section presents these test results.
Time sampling observation analysis. Screen-capture recordings of participants’
navigations were analyzed using two different time sampling observations: (1) observations of
the participants’ web-based actions, and (2) observations of participants’ web-based actions with
the audio recordings of participants’ think alouds.
1. Observations of web-based actions. Participants’ web-based actions were analyzed in
10-second intervals. This analysis had two purposes: (1) to track participants’ online
behaviours, and (2) to examine the differences in think aloud tasks across the three conditions.
Prior to this analysis, a list of web-based actions was created based on the website used in this
36
study (e.g., types of web pages, features, and web-based tools). Table 3.5 presents the list of 18
actions used for the first time sampling observation analysis.
Table 3.5
Time Sampling Analysis-Variables
Web-Based Action Description
1. Enters a search term Text is entered in search box
2. Maneuvers around virtual
tour
Moves around the virtual tour by clicking and dragging the
mouse
3. Opens content page Selects a content page from the tabs at top of the page (e.g.,
Teacher Information)
4. Opens food groups page Selects a food groups page from the tabs (e.g., Motivation for
Literacy)
5. Opens homepage Selects the homepage from the tabs at the top of the page
6. Opens how to videos Selects the how to video page from the tabs at the top of the page
7. Opens outside link Clicks on a hyperlink and views an outside website
8. Opens recipe Selects a recipe (a lesson)
9. Opens recipe finder page Selects the recipe finder page from the tabs at the top of the page
10. Opens virtual tour Selects and views a virtual tour
11. Selects filter option Topic choice and grade is narrowed down using the filter option
12. Starts a video Selects a video by clicking on the video image
13. Stops video before the end The video is stopped before the end
14. Takes a note Text is entered in a Word document
15. Uses virtual tour arrow Within the virtual tour, the arrows are selected to move around
the classroom
16. Uses virtual tour zoom Within the virtual tour, the zoom function is selected to view a
close up of student work and teacher materials
17. Views a photograph Within a recipe, selects the photo gallery to view photos
18. Views a video Time viewing videos is tracked in seconds
Participants’ navigations were observed and counts were made in 10-second intervals. A
highlighted cursor helped track participants’ actions during the screen recording observations.
37
Actions were tallied for each participant and frequency counts were used to compare the
conditions. Appendix B-3 presents a chart showing these frequency counts for each condition.
2. Observations of web-based actions with think aloud recordings. A second time
sampling analysis was performed to determine the correspondence between web-based actions
and verbalizations. Participants’ web-based actions were observed simultaneously with the
think aloud audio recordings. In 10-second intervals agreements and discrepancies between the
actions observed on the screen recordings and the think aloud verbalizations were recorded on a
chart. Appendix B-4 presents a sample chart from each condition. The results and discussion
section includes the findings of both time sampling observation analyses.
Interview analysis. The interview analysis involves three parts: (1) transcription, (2)
statistical analysis, and (3) a general inductive approach to analysis.
1. Transcription. Interview audio recordings were transcribed verbatim resulting in 45
interview transcripts. Once again, the primary researcher transcribed all 45 interviews. The
transcripts were entered into NVivo.
2. Statistical analysis. Numerical responses to interview question two, which asked
participants to rate their comfort level of thinking aloud on a 7-point scale, were entered into
SPSS for statistical analysis. ANOVA was performed on the participants’ comfort level ratings.
ANOVA results, including descriptive statistics, are discussed in the results and discussion
section.
3. General inductive approach to analysis. A qualitative analysis followed the ANOVA
procedure. The purpose of analyzing the interview responses qualitatively was to provide
possible reasons why participants were comfortable or uncomfortable with their experience of
thinking aloud. During data collection, participants were asked to provide a rationale for their
comfort level rating. These rationales were analyzed using a general inductive approach to
38
analysis, which allows themes to emerge from the data (Thomas, 2006). The descriptive
responses were first read reflectively. Phrases were then categorized and interconnected. This
resulted in six themes related to participants’ comfort with thinking aloud. The results and
discussion section presents the findings of the interview analysis.
Data Review Process
Variables used in the quantitative analyses were examined to determine if the
assumptions of normality and equal variances were met. Appendix B-5 summarizes the data
screening results including the descriptive statistics, normality tests, and homogeneity of
variance tests. The data examination revealed that variables were both normal and skewed
(positively or negatively). Skewed variables indicate an uneven distribution of scores above and
below the mean; therefore, both parametric and non-parametric tests were performed. The non-
parametric test used in this study was the Kruskal-Wallis test which is less restrictive than
ANOVA, its parametric counterpart (Field, 2009). Any variables that included extreme outliers
were also examined and reanalyzed with extreme outliers removed. According to the
questionnaire data, the extreme outliers did not demonstrate any unique characteristics that
required further examination. A comparison of ANOVA (with and without outliers) and
Kruskal-Wallis tests indicated similar results; therefore, the parametric test results are presented
for this study.
Results and Discussion
Given the complexity of study 1, this chapter organizes the results and discussion
together according to the three research questions. This section begins with a description of the
results from the think aloud analysis and discusses key findings that resulted from this analysis.
Next, this section presents the results from the time sampling analysis and discusses key
39
findings from the time sampling analysis. This section concludes with a description of the
interview analysis results and a discussion of the key findings from the interview analysis.
A general discussion of the findings follows the results and discussion section. The
general discussion of study 1 begins with a description of participants’ typical experience of
completing the website task and think aloud from each of the three conditions. These
descriptions offer a representation of participants’ experience with the three types of think aloud
methods. The general discussion also presents the benefits and limitations of each think aloud
method based on this study’s findings.
Research Question 1: To What Extent Do Participants’ Verbalizations Differ Across the
Three Think Aloud Methods?
This section is divided into three parts (see Figure 3.2). The first two parts present the
results and discussion of the think aloud analysis: the first will involve the thought unit variables
and the second will involve the word count and total number of thought unit variables. Both
will highlight the main effects, post hoc results, and significant findings. The third part of this
section presents Pearson correlations of the variables from the think aloud analysis.
Research Question 1: To what extent do participants’ verbalizations
differ across the three think aloud methods?
1. Thought unit variables
a. Main effects
b. Post hoc test results and discussion—significant findings
2. Word counts and number of thought units
a. Main effects
b. Post hoc test results and discussion—significant findings
3. Pearson correlations of think aloud variables
Figure 3.2. Topics pertaining to research question 1 results and discussion.
40
Thought unit variables. A one-way between subjects ANOVA was performed to
compare the percentage of each category verbalized by participants in each think aloud
condition. The analyses revealed a significant main effect for planning, F(2, 42) = 7.05, p =
.002, η2
= .251; connecting, F(2, 42) = 8.35, p = .002, η2
= .251; reasoning, F(2, 42) = 22.01, p
= .000, η2
= .512; and reflecting, F(2, 42) = 5.36, p = .008, η2
= .203. ANOVA results also
indicated significant main effects for describing website features, F(2, 42) =11.05, p = .000, η2
=
.345; and for reading, F(2, 42) =43.81, p = .000, η2
= .676. There were no significant main
effects for the other five categories. Table 3.6 displays the summary statistics for the main
effects and the means and standard deviations for each thought unit variable. What follows is a
discussion of each significant main effect and possible explanations for each finding. Direct
quotes from the think aloud transcripts are included to support the discussions.
Table 3.6
ANOVA Summary Statistics for Thought Units
Concurrent
Condition
Retrospective
Condition
Virtual Revisit
Condition
M
(SD)
M
(SD)
M
(SD)
F p η2
Planning
0.407 (0.70) 3.96 (3.83) 4.17 (3.66) 7.05 .002* .251
Connecting
6.04 (3.59) 13.23 (6.53) 9.08 (3.81) 8.35 .001* .285
Reasoning
2.23 (2.46) 10.99 (7.65) 19.27 (9.16) 22.01 .000** .512
Reflecting
16.71 (7.70) 26.86 (11.45) 25.08 (7.48) 5.36 .008* .203
Evaluating
Website
Content
10.87 (4.96)
12.18 (8.71)
15.36 (5.27)
1.87
.166
.082
Evaluating
User
Experience
7.96 (6.98)
12.13 (9.03)
7.47 (3.30)
2.09
.136
.091
Diversion
5.97 (3.01) 0.79 (2.08) 2.99 (3.42) 13.65 .067 .394
Understanding
5.99 (3.50) 2.97 (4.46) 3.36 (3.90) 2.88 .067 .121
41
Describing
Procedural
Behaviour
11.64 (6.21)
11.71 (9.12)
9.37 (4.91)
0.55
.582
.025
Describing
Website
Features
10.15 (4.87) 3.45 (4.83) 3.56 (3.53) 11.05 .000** .345
Reading 22.03 (12.79) 0.00 0.29 (0.64) 43.81 .000** .676
*p < .05, **p < .01
Note: df = (2, 42) for all variables
Planning. Post hoc comparisons using Tukey HSD revealed that participants in the
retrospective and virtual revisit conditions verbalized a significantly greater proportion of
planning thought units than participants in the concurrent condition (p < .05). One possible
reason for this finding relates to cognitive load. Cognitive load refers to the amount of mental
effort in working memory: cognitive load “represents the load that performing a particular task
imposes on the cognitive system” (Pass & Van Merrienboer, 1994, p. 122). Participants in the
concurrent condition were required to use a higher level of mental energy to process the given
information as they simultaneously completed the website task (Ping Lim, 2004); thus, the
cognitive load on their working memory may have diminished the quality of their
verbalizations. This is consistent with Ericsson and Simon (1993) who state that concurrent
reporting may become difficult to maintain under high cognitive load conditions. Cognitive
load research also indicates that during multimedia tasks, concurrent reporting will interfere
with information processing and limit the extent of the thought units (Nielsen, Clemmensen, &
Yssing, 2002; van Gog et al., 2009). As the participants in the concurrent condition verbalized
their thoughts while simultaneously completing the website task, the cognitive demand on their
working memory increased: their brains prioritized information processing over verbalizations.
Therefore, participants in the concurrent condition verbalized fewer planning thought units
(Cooke, 2010).
42
In contrast, participants in the retrospective and virtual revisit conditions had the
advantage of thinking aloud after the website task had been completed; thus, the cognitive load
on their working memory was lighter during their think aloud than those in the concurrent
condition. As a result of a lighter cognitive load, participants in the retrospective and virtual
revisit conditions had more cognitive resources available during the think aloud; therefore, they
could focus their mental energy on constructing and verbalizing complex ideas.
Planning is a complex cognitive activity and involves visualizing the future, producing
and generating new information, and “putting elements together to form an original product”
(Krathwohl, 2002, p. 215). Planning is also an indicator of teachers’ intentions for practice
(Clark & Peterson, 1986; Krathwohl, 2002). The planning thought units verbalized by
participants in the retrospective and virtual revisit conditions contained complex information
that demonstrate intentions for practice. For instance, a video on social studies and writing
integration led a participant in the retrospective condition “to think of a different idea that [she]
could bring into [her] class which includes doing cross-curricular work with probability and
ancient civilizations and trading cards and games.” Similarly, a teacher in the virtual revisit
condition described how she could modify a lesson to include cross-curricular integration:
“Building interviewing and report writing skills, I know that there is a lot that I can do with this
in terms of reading and writing and oral and drama and themes like social justice and history
and so on.” These examples demonstrate complex verbalizations; teachers are planning as they
use the professional development website—they are visualizing the future and beginning to
generate new ideas.
Connecting. Post hoc tests using Tukey HSD revealed that participants in the
retrospective condition verbalized a significantly greater proportion of connecting thought units
than participants in the concurrent condition (p = .001). No significant differences in
43
connecting thought units were found between the retrospective and virtual revisit conditions and
between the concurrent and virtual revisit conditions. As with planning, connecting is a
complex cognitive activity—connecting involves activating prior knowledge and experiences,
and recalling an idea, a concept or an event in relation to new information. Possible reasons for
the above findings can be drawn from research on information processing and memory recall.
Information processing is enhanced when new incoming information is connected to prior
knowledge and previous experiences (Mastin, 2010; Weber, Corrigan, Fornash, & Neupauer,
2003). Information processing is also enhanced when new material is interesting to the learner
(Garner & Gillingham, 1991). The more deeply new information is processed (i.e., through
connections and interest), the more likely it will be recalled. In contrast to participants in the
concurrent condition, participants in the retrospective condition were able to process
information during the website task on a much deeper level because they did not have the same
cognitive demands of verbalizing their thoughts while simultaneously completing the website
task. They were able to verbalize significantly more connections with past experiences and
interests than participants in the concurrent condition. For instance, a participant in the
retrospective condition connected components of the website to her current reading program:
After visiting a virtual tour, I was interested in going through all the different parts of the
balanced literacy and I clicked on a couple of things that interested me and that I’m
working on, like comprehension skills and fluency and word building.
As previously mentioned, the second possible reason for the above finding relates to
memory recall. Participants in the retrospective condition did not have access to a visual cue or
memory aid. It is possible that in the absence of a visual cue, participants in the retrospective
condition verbalized more connecting thought units because they recalled meaningful memories
established during the website task. For instance, a participant in the retrospective condition
44
connected website content on hand writing to a student in her class: “I saw something about
hand writing which made me think about a student that I have in grade one who…it’s quite a
struggle for her to read what she’s writing.” The information that this participant recalled was
meaningful to her because it directly related to the needs of a student in her classroom.
Reasoning. Post hoc tests using Tukey HSD revealed that participants in the virtual
revisit condition verbalized a significantly greater proportion of reasoning thought units than
participants in the other two conditions (p < .05). Reasoning involves justifications for
decisions. Participants who verbalized reasoning thought units provided a rationale for their
web-based actions and behaviour. Two possible reasons for the above finding relate to
cognitive load and access to a visual aid. First, in contrast to participants in the concurrent
condition, participants in the virtual revisit condition had a lighter cognitive load on their
working memory. As previously discussed, this lighter cognitive load freed cognitive resources
and allowed participants in the virtual revisit condition to explain their web-based actions and
navigational choices. Secondly, in contrast to participants in the retrospective condition,
participants in the virtual revisit condition had direct access to their web-based actions via the
screen-capture recording. The screen recording captured participants’ website navigation and
acted as a visual aid during the think aloud. Participants in the virtual revisit condition utilized
the visual information as an aid to recall their navigational decisions and why they made them.
Taken together, available cognitive resources and direct access to web-based actions allowed
participants in the virtual revisit condition to extend their descriptions and clarify the reasons for
their navigational choices.
Reasoning about behaviour moves beyond simple descriptions of actions and offers rich
explanations about decisions. Three examples from the virtual revisit condition demonstrate
this perspective:
45
(1) A participant provided a descriptive rationale for her decision to remain on a
particular webpage for an extended period of time: “At this point I was trying to just read
what the student wrote to get an idea of whether they were creating their own stories or
whether they were doing more of a retell”;
(2) A participant clarifies her thinking about why she viewed just one virtual classroom
by explaining: “I only managed to look at the one teacher’s classroom because there was
actually a lot of information to be gained from her”;
(3) A participant in the virtual revisit condition extended the simple statement: “I ended
up choosing reading comprehension strategies,” by offering a reason for her choice: “I
was considering looking at text structures and I ended up choosing reading
comprehension strategies because I’m trying to make that one of the main focuses of our
reading program.”
Reasoning thought units provide thorough descriptions, clarifications, extensions and
overall greater insight into participants’ navigational choices—participants explain why they
make particular navigational decisions.
Reflecting. Post hoc tests using Tukey HSD revealed that participants in both the
retrospective and virtual revisit conditions verbalized a significantly greater proportion of
reflecting thought units than participants in the concurrent condition (p < .05). Along with the
explanation that relates to available cognitive resources, as previously described, a second
possible reason for this finding relates to the time lapse between the website task and the think
aloud. Participants in the retrospective and virtual revisit conditions completed the think aloud
immediately following the website task. Although relatively short, the time lapse between the
website task and think aloud may have allowed participants in the retrospective and virtual
revisit conditions to access meta-cognitive information from long term memory (Gaissmaier et
46
al., 2010). Participants in these two conditions reflected on their learning processes and how
they used the given website. Two examples from the retrospective and virtual revisit think
alouds demonstrate the types of reflective thoughts units verbalized by participants in these two
conditions:
(1) A participant in the retrospective condition reflected on her learning style, “I like
learning in that manner—having the visual to see what people are doing in their
classroom”;
(2) A participant in the virtual revisit condition reflected on her general use of online
professional resources: “When I look at websites, often I’m trying to look for specific
ideas, anchor charts or teaching strategies that I can take away right away and start trying
to use specifically on themes that I may not have covered.”
Describing website features and reading. Post hoc tests using Tukey HSD revealed that
participants in the concurrent condition verbalized a significantly greater proportion of
describing website features thought units and reading thought units than participants in the other
two conditions (p < .05). These findings are consistent with prior studies utilizing the
concurrent think aloud (McDonald et al., 2012; Zhao & McDonald, 2010). For instance, Zhao
and McDonald (2010) found that the categories containing the greatest number of utterances
during a concurrent think aloud were those relating to descriptive information and on-screen
text.
One possible reason for these findings is that the cognitive load diminished the quality of
verbalizations. As previously discussed, participants in the concurrent condition were required
to use a higher level of mental energy to process the given information as they simultaneously
completed the website task (Ping Lim, 2004). As participants in the concurrent condition
processed information from the given website, cognitive resources were less available for the
47
verbalization of complex thought processes. Therefore, participants in the concurrent condition
verbalized a greater proportion of thought units related to descriptions of website features and
reading text on the screen since these types of thought units are not as cognitively demanding to
verbalize.
Two quotes from participants in the concurrent condition demonstrate the simplicity of
the cognitive processes, describing website features and reading. First, a participant identifies a
pyramid image: “So here, reading pyramid.” The given website provides a reading pyramid as a
way for its viewers to understand and support reading development; however, this participant
merely describes the image as a reading pyramid and does not provide any further information.
Secondly, as a participant in the concurrent condition begins the website navigation, she reads
the main heading on the homepage: “Balanced literacy is a framework that presents literacy
concepts.” This participant moves on to read the next heading without providing further
thoughts about the main website heading.
With respect to the remaining thought units, findings indicated no significant differences
between the three conditions for evaluating, diversion, understanding, and describing procedural
behaviours. This suggests that regardless of the type of think aloud employed, participants will
verbalize a relatively equal number of thoughts related to evaluating the website, to their
confusion and understanding of the web-based tools and information, and to descriptions of their
own actions and online behaviours.
Word counts and total number of thought units. A one-way between subjects
ANOVA was performed to compare the frequency of words and thought units verbalized in
each think aloud condition. The summary statistics for the main effects and the means and
standard deviations are summarized in Table 3.7. The following section discusses the
significant main effects and possible explanations for each finding.
48
Table 3.7
ANOVA Summary Statistics for Word Count and Number of Thought Units
Concurrent
Condition
Retrospective
Condition
Virtual
Revisit
Condition
M
(SD)
M
(SD)
M
(SD)
F p η2
Word Count
1676.13
(394.89)
658.80
(229.00)
2637.87
(359.94)
130.421 .000** .861
Number of Thought
Units
147.40
(35.83)
30.73
(13.15)
123.27
(20.58)
90.756 .000** .812
**p < .01
Note: df = (2, 42) for all variables
The analysis revealed a significant main effect for both variables across the three
conditions: F(2, 42) = 130.42, p = .000, η2
= .861 for word count; and F(2, 42) = 90.76, p =
.000, η2
= .812 for number of thought units. As Table 3.7 shows, participants in the
retrospective condition verbalized the fewest number of words (M = 658.80, SD = 229.00) and
thought units (M = 30.73, SD = 13.15). Participants in the concurrent condition verbalized more
than twice the number of words as participants in the retrospective condition (M = 1676.13, SD
= 394.89) and more than four times the number of thought units than participants in the
retrospective condition (M = 147.40, SD = 35.83). Participants in the virtual revisit condition
verbalized the most number of words (M = 2637.87, SD = 359.94). However, participants in the
virtual revisit condition verbalized less thought units than teachers in the concurrent condition
(M = 123.27, SD = 20.58). This finding suggests that thought units verbalized by teachers in the
virtual revisit condition contained more words than thought units verbalized by teachers in the
concurrent condition.
Post hoc comparisons using Tukey HSD revealed significant differences in the
frequency of words (p < .01) and the frequency of thought units (p < .05) between all three
conditions. In contrast to the participants in the concurrent and virtual revisit conditions who
49
were asked to verbalize their thoughts for 20 minutes, participants in the retrospective condition
were not given a time constraint to verbalize their thoughts. Although an unlimited amount of
time was given to participants in the retrospective condition, these participants verbalized their
thoughts for an average four minutes 30 seconds. Therefore, it is not surprising that participants
in the retrospective condition verbalized significantly fewer words and thought units than the
other two conditions. A more interesting finding is the difference in word counts and number of
thought units between the concurrent and virtual revisit conditions. Participants in the virtual
revisit condition verbalized significantly more words than participants in the concurrent
condition; however, the reverse is true for the number of thought units. This finding indicates
that thought units produced by participants in the virtual revisit condition contained a greater
number of words. This suggests that the thought units verbalized by participants in the virtual
revisit condition were more complex than the thought units verbalized by participants in the
concurrent condition. For example, as a participant from the concurrent condition views the
homepage for the first time she describes her procedural behaviour: “I’m going to have a look at
the How To Videos.” A participant from the virtual revisit condition who also views the
homepage for the first time goes further to provide a reason for her behaviour: “I’m always
interested in ways to increase my students’ background knowledge of vocabulary and
comprehension so I go back and forth between comprehension and vocabulary before I
narrowed it down and selected vocabulary.” Both of these thought units were verbalized during
the participants’ initial view of the home page. However, the thought unit verbalized by the
participant in the virtual revisit condition is more complex in that it provides a reason for her
navigational choice.
Pearson correlations. Pearson correlations were computed to determine three
relationships: between the cognitive processes, between the frequency of word counts and
50
thought units, and between the cognitive processes and word counts and thought units (see
Table 3.8). According to the results, higher order cognitive processes (planning, connecting,
reasoning, and reflecting) were positively correlated to other higher order cognitive processes,
and negatively correlated to lower order cognitive processes (diversion, understanding,
describing website features, and reading). Similarly, lower order cognitive processes were more
likely to be positively correlated to other lower order cognitive processes and negatively related
to higher order cognitive processes. For instance, reading was negatively correlated to planning
(r = -.42, p < .01), connecting (r = -.39, p < .01), reasoning (r = -.50, p < .01), reflecting (r = -
.55, p < .01), evaluating website content (r = -.31, p < .05), and evaluating user experience (r = -
.37, p <.05). In contrast, reading was positively related to diversion (r = .46, p < .01),
understanding (r = .30, p < .05), and describing website features (r = .45, p < .01). These
findings corroborate the above ANOVA results and suggest that participants in the concurrent
condition, who were more likely to read text during the think aloud, were less likely to verbalize
thought units related to higher order cognitive processes.
Pearson correlations also revealed a positive correlation between the frequency of words
and the frequency of thought units (r = .70, p < .01). This finding indicates that as the number
of words increased the number of thought units increased. Results also revealed a negative
correlation between higher order cognitive processes and the number of thought units
verbalized. This suggests that teachers who verbalized thought units related to higher order
processes were more likely to verbalize fewer thought units than teachers who verbalized
thought units related to lower order processes.
51
Table 3.8 Pearson Correlations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Cognitive
Processes
1. Planning 1
2. Connecting .31* 1
3. Reasoning .36* .06 1
4. Reflecting .19 .38* -.10 1
5. Evaluating
Website
-.23 .03 .03 .23 1
6. Evaluating
Experience
.05 -.08 -.09 .16 -.34* 1
7. Diversion -.36* -.39** -.22 -.46** -.17 -.14 1
8.Understanding -.26 -.35* -.21 -.29 -.34* .05 .38* 1
9. Describing
Behaviour
-.12 -.20 .03 -.28 -.56** -.47** -.003 .01 1
10. Describing
Website
-.43** -.33* -.45** -.31* -.13 -.23 .39** .23 .10 1
11. Reading -.42** -.39** -.50** -.55** -.31* -.37* .46** .30* .18 .45** 1
12.Word
Count
.07 -.26 .28 -.01 .18 -.19 .25 .05 -.16 .02 -.05 1
13. Thought
Units
-.31* -.53** -.15 -.38** -.04 -.30* .57** .30* .01 .44** .54** .70** 1
14. Time on
Page
.30* .47* .12 .38* -.06 .04 -.36* -.11 .02 -.22 -.41** -.07 -.43** 1
15. Number of
Pages
-.21 -.34* -.27 -.37* -.29 -.08 .46** .46** -.04 .44** .50** .11 .55** -.68** 1
16. Comfort
Level
.11 .23 .28 .13 .21 -.15 -.25 .07 .11 -.37* -.34* .14 -.13 .05 -.17 1
*p < .05, **p < .01
52
Summary of Results and Discussion for Research Question 1
This section identified and discussed significant findings in relation to research question
1: To what extent do participants’ verbalizations differ across the three think aloud methods?
First, this section identified differences in the types of verbalizations produced by participants in
each condition. Participants in the retrospective and virtual revisit conditions verbalized
significantly more thought units related to higher order cognitive processes: planning,
connecting, reasoning, and reflecting. In contrast, participants in the concurrent condition
verbalized significantly more thought units related to lower order cognitive processes:
describing website features and reading. Possible reasons for these findings relate to cognitive
load on working memory, information processing, memory recall, visual aids, and the time lapse
between the website task and the think aloud.
This section also identified differences in the frequency of words and thought units
across the three conditions. Specifically, participants in the virtual revisit condition verbalized
the highest number of words and produced complex thought units. Results of the correlation
analysis indicated correlations between the cognitive processes and frequency of word counts
and thought units. The next main section presents the results and discussion of the time
sampling observation analysis.
Research Question 2: How Do Think Aloud Tasks Differ Across the Three Think Aloud
Conditions?
This section continues the results and discussion of study 1 by presenting the results of
the time sampling observation analysis in two parts (see Figure 3.3). The first part presents the
results and discussion of the time sampling observation analysis involving the web-based action
variables, and highlights the main effects and post hoc results, followed by a discussion of the
overall findings. The second part presents the results and discussion of the time sampling
53
observation analysis involving the web-based actions with the corresponding think aloud audio
recordings.
Research Question 2: How do think aloud tasks differ across the
three think aloud conditions?
1. Web-based action variables
a. Main effects
b. Post hoc results and discussion—significant findings
2. Web-based actions with the corresponding think aloud recordings
Figure 3.3. Topics pertaining to research question 2 results and discussion.
Web-based actions. A one-way between subjects ANOVA was performed on the web-
based action variables. The analysis revealed seven significant main effects: Time spent on
pages, F(2, 42) = 7.16, p = .002, η2
= .254; total number of pages visited, F(2, 42) = 11.33, p =
.000, η2
= .351; total number of food group pages visited, F(2, 42) = 8.50, p = .001, η2 = .288;
total number of content pages visited, F(2, 42) = 3.66, p = .034, η2 = .149; number of visits to
the homepage, F(2, 42) = 9.34, p = .000, η2 = .308; time participants spent viewing videos (in
seconds), F(2, 42) = 5.13, p = .010, η2
= .196; and number of videos that participants stopped
midway, F(2, 42) = 14.64, p = .000, η2
= .417. The analysis revealed no significant main effects
for the remaining variables. Table 3.9 summarizes the means, standard deviations, and main
effects for the web-based action variables. The remaining section presents the post hoc results
for the significant findings. This is followed by a discussion of the overall results of the time
sampling observation analysis involving the web-based actions.
54
Table 3.9
ANOVA Summary Statistics for Web-Based Action Variables
Concurrent
Condition
Retrospective
Condition
Virtual Revisit
Condition
M
(SD)
M
(SD)
M
(SD)
F P η2
Time on Each
Page (in
seconds)
36.07
(9.08)
52.53
(15.11)
47.80
(11.87)
7.16 .002* .254
Number of
Pages Visited
26.93
(9.02)
16.13
(4.19)
18.60
(5.32)
11.33 .000** .351
Number of Food
Group Pages
Visited
6.40
(4.12)
2.80
(1.21)
2.93
(1.92)
8.50 .001* .288
Number of
Content Pages
Visited
1.60
(1.88)
0.87
(.834)
0.40
(.507)
3.66 .034* .149
Visits to
homepage
2.13
(1.60)
0.60
(.507)
0.67
(.90)
9.34 .000** .308
Time Viewing
Videos (in
seconds)
150.67
(125.10)
334.00
(235.00)
355.33
(200.32)
5.13 .010* .196
Number of
Videos Stopped
Midway
4.27
(2.05)
1.20
(1.26)
2.53
(2.51)
7.26 .002* .257
Enters a Search
Term
1.13
(1.96)
0.40
(.83)
0.40
(.74)
1.59 .216 .070
Maneuvers
Around Virtual
Tour
0.93
(1.71)
0.80
(2.34)
0.67
(1.50)
.08 .928 .004
Visits to How-
To Videos
1.00
(.845)
0.47
(.516)
0.53
(.743)
2.48 .096 .106
Visits to Outside
Link
2.13
(2.95)
0.80
(1.15)
1.13
(1.60)
1.73 .191 .076
Views Recipe
Page
6.80
(3.17)
6.13
(3.98)
6.27
(2.87)
0.16 .849 .008
Views Recipe
Finder Page
4.27
(2.89)
3.20
(3.84)
5.20
(2.83)
1.45 .247 .065
55
Visits Virtual
Tour
1.67
(1.45)
1.20
(.941)
1.33
(.617)
0.77 .468 .036
Selects Filter
Option
1.20
(1.66)
0.87
(2.13)
1.13
(1.64)
0.14 .870 .007
Takes a Note 0.00 0.60
(2.06)
1.53
(3.18)
1.87 .167 .082
Uses Virtual
Tour Arrow
2.00
(3.12)
1.53
(2.42)
2.40
(4.07)
0.26 .769 .012
Uses Virtual
Tour Zoom
0.07
(0.26)
0.53
(0.92)
0.20
(0.56)
2.13 .131 .092
Views a
Photograph
5.73
(5.44)
8.07
(6.20)
6.20
(8.26)
0.33 .724 .015
*p < .05, **p < .01
Note: df = (2, 42) for all variables
Time spent on pages and total number of pages visited. Post hoc comparisons using
Tukey HSD revealed that participants in the retrospective and virtual revisit conditions spent
significantly more time on web pages than participants in the concurrent condition (p < .05). In
addition, participants in the retrospective and virtual revisit conditions visited significantly
fewer pages than participants in the concurrent condition (p < .05). Pearson correlation results
support the ANOVA results and show a negative correlation between the time spent on pages
and the number of pages visited (r = -.68, p < .01). This finding confirms that participants who
spent more time on web pages visited fewer web pages during their 20-minute navigation.
Pearson correlation results also show a positive correlation between the time spent on web pages
and planning, connecting, and reflecting thought units.
Total number of food group pages, content pages, and homepage visits. Post hoc
comparisons using Tukey HSD revealed participants in the concurrent condition viewed
significantly more food groups pages than the retrospective condition (p = .002), and virtual
revisit condition (p = .003), while no significant differences were found between the
retrospective and virtual revisit conditions (p = .990). Additionally, participants in the
56
concurrent condition viewed significantly more content pages than the virtual revisit condition
(p = .027), while there were no significant differences between the retrospective and concurrent
conditions (p = .240) and between the retrospective and virtual revisit conditions (p = .554).
Finally, participants in the concurrent think aloud condition viewed the homepage significantly
more times than participants in the retrospective condition (p = .001) and the virtual revisit
condition (p = .002), while no significant difference was found between the retrospective and
virtual revisit conditions (p = .985).
Video viewing. Post hoc comparisons using Tukey HSD showed that participants in the
retrospective condition spent significantly more time viewing videos than participants in the
concurrent condition (p = .033) and also stopped fewer videos midway than participants in the
concurrent condition (p = .000). Similarly, participants in the virtual revisit condition spent
significantly more time viewing videos than teachers in the concurrent condition (p = .015) and
stopped fewer videos midway than teachers in the concurrent condition (p = .001). There were
no significant differences in time spent viewing videos and the number of videos stopped
midway between the retrospective and virtual revisit conditions. These findings indicate that
participants in the retrospective and virtual revisit conditions spent more time viewing videos
than participants in the concurrent condition.
Taken together, the overall results of the first time sampling observation analysis
involving the web-based actions suggest that participants in the retrospective and virtual revisit
conditions had higher levels of engagement with the information, images, tools, and videos on
each page of the website during their 20-minute navigation than participants in the concurrent
condition. One possible reason relates to the dual activity of completing the website task while
simultaneously thinking aloud. As previously described, the heavy cognitive load on working
memory disrupted both concurrent participants’ verbalizations and their task performance.
57
Efforts made to verbalize thoughts may have changed the way the participants in the concurrent
condition used the website (Guan et al., 2006). In contrast, participants in the retrospective and
virtual revisit conditions were able to focus exclusively on their navigation without the
additional activity of simultaneously verbalizing their thoughts.
A second possible reason for the differences in web-based actions relates to the stage in
which the instructions were given to participants. Prior to completing the website task,
participants in the retrospective and virtual revisit conditions were only given the first part of the
instructions: to use the website as they normally would when seeking information related to
their practice. In contrast, participants in the concurrent condition were given the web task
instruction as well as the think aloud instruction. Despite the practice think aloud that
participants in the concurrent condition completed prior to the study think aloud, it appears that
the preconceived notion of the think aloud task may have interfered with the concurrent
participants’ web navigation. As suggested by Ericsson and Moxley (2011), instructing the
participants to think aloud while simultaneously using the website, may have a negative effect
on participants’ task performance. Reactivity, “influences of the verbalizations on the decision
process” (Ranyard & Svenson, 2010, p. 119), appeared to negatively affect the concurrent
condition—participants may have altered their navigation due to their awareness of the task.
Web-based actions with the corresponding think aloud recordings. A second time
sampling analysis using the screen-capture recordings and think aloud audio recordings was
conducted to determine the correspondence between participants’ actions and verbalizations. In
general, participants in the retrospective condition verbalized their thoughts for an average of 4
minutes 30 seconds with a range of 1:59-6:52. In contrast, all but two participants in the other
two conditions verbalized their thoughts for 20 minutes. The two participants who did not
verbalize their thoughts for 20 minutes were from the concurrent condition and requested to stop
58
the think aloud at 18 minutes and at 18 minutes 40 seconds respectively. The final
verbalizations from these two participants were: “What else can I say? I’m not exactly sure.
Let’s stop there,” and “I’m done talking, I think.”
Results of this second time sampling analysis revealed the correspondences between
web-based actions and verbalizations. Agreements, discrepancies, and silent intervals are
reported for each condition.
Concurrent condition. During the 10-second intervals, agreements between the web-
based actions and verbalizations were recorded 93.5% of the time for participants in the
concurrent condition. That is, the action that was observed on the screen recording directly
corresponded with the verbalization. Discrepancies accounted for 6.5% of the time and
included intervals of silence and participants’ request to end the website task and think aloud
early. Intervals of silence occurred for 14 participants in the concurrent condition and
accounted for 5.4% of the total time. These intervals of silence occurred during a video. In
addition, two participants ended their think aloud early (accounting for 1.1%).
Retrospective condition. Analysis of the retrospective condition showed agreements
between web-based actions and verbalizations 17.3% of the total time and discrepancies
between participants’ actions and verbalizations 82.69% of the total time. Discrepancies were
found to occur between the first action and the first verbalization. This type of discrepancy
occurred for seven participants assigned to the retrospective condition (47%). For example, the
first verbalization from one participant was, “I think I went to year one first.” This verbalization
was discrepant with this participant’s first web-based action: the screen recording showed this
participant opening a content page and then navigating to a food group page. Another
participant in the retrospective condition stated that she immediately navigated to a web page on
vocabulary development; however, this participant initially spends time on the homepage, then
59
visits the introductory page and views two videos. She finally navigates to a web page on
vocabulary at 3 minutes 30 seconds into her navigation. The think aloud omitted the first three
main actions of her web navigation. As a further example, a participant began her think aloud
by stating, “I went to the recipes right away;” however, observations of the screen recording
show this participant initially entering a virtual classroom tour for grades one and two.
With respect to the eight participants in the retrospective condition whose first action did
correspond to their first verbalization (53%), discrepancies appeared, on average, at 49 seconds
into the think aloud. Omissions were most often observed. For example, a screen recording
shows one participant in the retrospective condition taking notes in a Word document during her
20-minute navigation. Her first note occurs at 6 minutes 20 seconds followed by seven
additional notes during the 20-minute navigation; however this participant omits this encoding
strategy entirely during her think aloud. Another participant in the retrospective condition visits
content pages and virtual tours during her navigation; however omits these actions during her
think aloud. Similarly, a participant in the retrospective condition is observed viewing
information related to assessment and writing conventions for a total of 10 minutes; however
this participant omits this information during her think aloud. The above finding is consistent
with Kuusela and Paul’s (2000) study which found retrospective reports to be more prone to
errors of omission. One possible reason that participants omitted information from their think
aloud relates to short term memory recall. Participants in the retrospective condition recalled
information without a memory aid. In the absence of a memory aid, think alouds were
incomplete (Branch, 2006).
Web-based actions that were correctly recalled occurred at the beginning of the 20-
minute website task and towards the end of the 20-minute website task. This finding is
consistent with the serial position effect—items most likely recalled from a series tend to be
60
those items from the beginning or end of a given list (Ebbinghaus, 1964). Participants in the
retrospective condition were more likely to recall their first action and their most recent action
from the website navigation. Many of their intermediate web-based actions were omitted from
their think aloud.
Finally, the second time sampling analysis revealed participants’ difficulties in recalling
information. Nine retrospective participants (60%) expressed these difficulties during the think
aloud. For example, at two minutes 56 seconds into the think aloud one participant in the
retrospective condition asked, “What else did I do?” At three minutes 43 seconds, another
participant stated, “I'm trying to think what else.” Similarly, towards the end of their think
alouds, participants in the retrospective condition asked themselves, “What else did I look at?”
“What else did I do” and “What else was I thinking?” On average, these nine participants
verbalized their difficulties in recalling their behaviours at two minutes 58 seconds into their
think aloud.
Virtual revisit condition. Agreements between web-based actions and verbalizations
were recorded for the virtual revisit condition 99.2% of the time. The analysis revealed
intervals of silence as a video played for two participants in the virtual revisit condition,
accounting for 0.8% of the total time.
Summary of Results and Discussion for Research Question 2
This section identified and discussed significant findings pertaining to research question
2: How do think aloud tasks differ across the three think aloud conditions? Taken together, the
results of both time sampling observation analyses reveal differences in the think aloud tasks
across the three conditions. Specifically, participants across the conditions differed with respect
to the time spent on each web page, the number of visits made to web pages, the types of pages
they visited most often, the time spent viewing videos, and the number of videos that were
61
stopped before the end. In addition, differences were found in the correspondences between the
web-based actions and think aloud audio recordings across the three conditions. Specifically,
discrepancies between web-based actions and verbalizations occurred most often in the
retrospective condition. The next main section presents the results and discussion of the
interview analysis.
Research Question 3: To What Extent Does Comfort Level with Thinking Aloud Differ
Across the Three Think Aloud Methods?
This section continues the results and discussion of study 1 by presenting the results of
the interview analysis in two parts (see Figure 3.4). The first part presents the results and
discussion of the interview analysis involving the reported comfort levels with thinking aloud,
and highlights the main effects, post hoc results, and significant findings. The second part of
this section shares the qualitative findings of the interview analysis and discusses possible
reasons for participants’ comfort level ratings.
Research Question 3: To what extent does comfort level with
thinking aloud differ across the three think aloud methods?
1. Comfort level with thinking aloud
a. Main effect
b. Post hoc results and discussion—significant findings
2. Qualitative findings and discussion
Figure 3.4. Topics pertaining to research question 3 results and discussion.
Comfort level with thinking aloud. A one-way between subjects ANOVA was
performed to compare the comfort levels of thinking aloud reported by the participants during
the post-task interview. Participants were asked to rate their comfort level with thinking aloud
on a 7-point scale (1 indicating very uncomfortable to 7 indicating very comfortable). On
average, participants in the concurrent condition reported their comfort level as 5.07 (SD =
62
1.22), participants in the retrospective condition reported their comfort level as 6.07 (SD =
0.96), and participants in the virtual revisit condition reported their comfort level as 6.27 (SD =
0.88). The main effect for comfort level reported by participants was significant, F(2, 42) =
5.81, p = .006, η2
= .217. Post hoc comparisons using Tukey HSD indicated that the participants
in the retrospective condition felt significantly more comfortable thinking aloud than
participants in the concurrent condition (p = .030). Participants in the virtual revisit think aloud
condition also felt significantly more comfortable thinking aloud than participants in the
concurrent think aloud condition (p = .008). No significant differences were found between the
retrospective and virtual revisit conditions (p = .857). These results indicate that participants in
the retrospective and virtual revisit conditions felt significantly more comfortable thinking aloud
than participants in the concurrent condition.
Pearson correlation results were computed between reported comfort levels and the
cognitive processes. Results revealed negative correlations between comfort level and
describing website features (r = -.37, p < .05) and comfort level and reading (r = -.34, p < .05).
These results corroborate the ANOVA findings, and suggest that participants in the concurrent
condition, who described website features and read most often during their think aloud, felt the
least comfortable with the process of thinking aloud.
Given the previous discussion related to cognitive load, it is not surprising that
participants in the concurrent condition rated their comfort level significantly lower than
participants in the other two conditions. Participants in the concurrent condition had the added
demand of completing the website task during their think aloud. Factors causing reactivity may
have also influenced participants’ comfort level. For instance, auditory feedback and task
instructions could have interfered with concurrent participants’ comfort level with thinking
aloud. This is consistent with Van den Haak and her colleagues’ findings (2003) that the
63
concurrent think aloud may result in greater reactivity and therefore greater discomfort with the
process of thinking aloud. The next section provides additional reasons for differences in
comfort level with thinking aloud across the three conditions.
Qualitative findings and discussion. As previously mentioned, interview responses
were further examined qualitatively. The purpose of performing this qualitative analysis was to
provide possible reasons for participants’ reported comfort levels. A general inductive approach
to analysis resulted in six themes related to participants’ reported comfort level: (1) Self-
awareness, (2) Negative feelings, (3) Memory retrieval, (4) Thinking aloud over time, (5)
Classroom experience, and (6) Benefits of thinking aloud. Table 3.10 summarizes each theme.
The remaining section discusses each theme and includes selected quotes to support each theme.
Table 3.10
Themes Related to Reported Comfort Level
Theme Description
1. Self-Aware An awareness of their experience and the study task
2. Negative Feelings Negative feelings of distractedness, awkwardness, and stress
3. Memory Retrieval Difficulties recalling actions and thoughts
4. Thinking Aloud Over Time Referring to an increase in comfort level over time during the
think aloud
5. Classroom Experience Use of think alouds in the classroom
6. Benefits of Thinking Aloud The value of thinking aloud as a teaching and learning tool
Theme 1: Self-awareness. Five participants in the concurrent condition (33%) described
how they were conscious of the think aloud experience during the website task. For instance,
one participant made an explicit reference to the study task and awareness of time: “I was
uncomfortable because I was conscious of time and whether I would have things to say after a
64
while.” Another participant in the concurrent condition reported a level 3 “because I felt like I
knew that someone was listening so that kind of makes it uncomfortable for me personally.”
Theme 2: Negative feelings. Ten total participants (22%) who rated their comfort level
as five or less described how thinking aloud contributed to negative feelings of distractedness,
awkwardness, and stress (concurrent, n = 4; retrospective, n = 5; virtual revisit, n = 2). One
participant in the concurrent condition stated that thinking aloud is “not something I really do…I
found myself distracted, reading to myself half the time.” Another participant in the concurrent
condition felt that “it was just kind of strange…I’m never really aware of what I’m thinking so
to have to verbalize it you have this metacognitive piece that’s not usually there.” Participants
in the retrospective condition also described their negative feelings towards thinking aloud. For
instance, one participant described how being conscious of her own voice contributed to
negative feelings: “I don’t really like talking out loud and hearing my own voice speak of
things.” Another participant in the retrospective condition described how thinking aloud was a
stressful experience and that a visual aid would have eased her stress: “It can be stressful, it’s
hard to put someone on the spot like that…I think thinking aloud for me is easier if I have some
type of visual to go along with it.” In addition, a participant in the virtual revisit condition felt
that thinking aloud was “a little unnatural, so it feels weird.”
Theme 3: Memory retrieval. Four participants in the retrospective condition (27%)
referred to the difficulty of recalling their web-based actions. For instance, one participant
reported that “sometimes it’s hard to remember what you did…it’s easier to think aloud in the
moment as opposed to thinking aloud about something that just happened.” Another participant
stated: “I was worried that I was not saying it in the right order and that I didn’t remember what
I did.”
65
Theme 4: Thinking aloud over time. Six of the total participants (13%) stated that as
the think aloud continued they became more comfortable with the process (concurrent, n = 2;
retrospective, n = 1; virtual revisit, n = 3). For instance, a participant in the concurrent condition
stated: “It was definitely an awkward experience but it was getting better as I was going through
it.” Another concurrent participant said, “Once I got going [thinking aloud] was fine.” A
participant in the retrospective condition thought thinking aloud was “a little bit weird to be
doing, but then once you get started you’re fine.” Similarly, a participant in the virtual revisit
condition shared: “At first I was a little bit worried—would I remember what I was thinking
about 20 minutes ago?” This same participant continued by stating:
But when I saw where my mouse was headed and where I was scrolling up and down I
remembered pretty clearly and easily what I was thinking about at those particular times
so it wasn’t quite as hard to remember as I thought.
Finally, a teacher in the virtual revisit condition who rated her comfort level as a 7 stated: “I
think people get better at it over time and I think I’ve improved in the ability to think aloud by
doing it.”
Theme 5: Classroom experience. Sixteen of the total participants (36%) connected their
comfort level to their teaching experience (concurrent, n = 3; retrospective, n = 7; virtual revisit,
n = 6). These participants referred to the teaching strategy, modeling thinking aloud. Their
prior experience with this strategy in the classroom contributed to a higher level of comfort
during the think aloud. For instance, a participant in the retrospective condition who rated her
comfort level as a 7 stated: “We’re taught to think aloud in front of the students all the time, so
when we think aloud they think about how their thinking processes could be, so we’re supposed
to model all that.” Another participant in the retrospective condition said: “I spend my day in
front of children being really silly, so [thinking aloud] was fine.” A participant in the virtual
66
revisit condition referred to her experience of thinking aloud during her reading program: “I
definitely do it with my children all the time, when reading stories. So I think it’s really valuable
as a comprehension strategy and gets you thinking why you do what you do.” Similarly, a
participant in the virtual revisit condition said:
I’m used to thinking aloud in front of my students. I do think alouds a lot and I
remembered so much of what I was thinking that it wasn’t hard for me…I was kind of
surprised…just seeing the image again watching the cursor. It was really the cursor that
was the connector to my thoughts.
Finally, a participant in the virtual revisit condition suggested that, “as a teacher you get very
accustomed to that concept of thinking aloud.”
Theme 6: Benefits of thinking aloud. Seventeen of the total participants (38%)
described benefits of thinking aloud (concurrent, n = 6; retrospective, n = 4; virtual revisit, n =
7). For example, one participant in the retrospective condition suggested that thinking aloud
could be “a valuable tool for learning…it could be for children or even adults to consolidate
what you’ve seen.” Furthermore, a participant in the virtual revisit condition “thought it was
really interesting” and began to think about “how I could use it with the kids and where it would
be valuable.”
Summary of Results and Discussion for Research Question 3
This section identified and discussed the significant findings pertaining to research
question 3: To what extent does comfort level with thinking aloud differ across the three think
aloud methods? Findings revealed differences in reported comfort level across the three
conditions. Participants in the retrospective and virtual revisit conditions reported feeling more
comfortable with the process of thinking aloud than participants in the concurrent condition.
67
This section also identified possible reasons for participants’ comfort level ratings. Cognitive
load and reactivity were discussed as possible reasons why participants in the concurrent
condition felt less comfortable with thinking aloud than those in the other two conditions. In
addition, the qualitative analysis revealed seven themes related to reasons why participants felt
comfortable or uncomfortable with thinking aloud: self-awareness, negative feelings, memory
retrieval, thinking aloud over time, and classroom experience. The qualitative findings also
showed benefits of thinking aloud as a teaching and learning strategy.
General Discussion of Study 1
The previous section presented the results and discussion of the three analyses: think
aloud analysis, time sampling analysis, and interview analysis. Based on the results and
discussion, this section presents the general discussion of study 1 in two parts (see Figure 3.5).
The general discussion begins with descriptions of participants’ typical experiences with each
think aloud condition. The first description includes the website task and the think aloud for the
concurrent condition. The second description includes the website task for both the
retrospective and virtual revisit conditions. The third description includes the think aloud for
the retrospective condition. The final description includes the think aloud for the virtual revisit
condition. These descriptions are representations of participants’ experiences and include
characteristic components of each think aloud condition based on the findings of study 1. The
purpose of including the characteristic components of each think aloud is to present a theoretical
portrait of how participants experienced each think aloud condition.
The second part of the general discussion highlights the benefits and limitations of
employing each think aloud method based on the findings of study 1. Chapter 3 ends with study
1 limitations, and significance and implications. Figure 3.5 outlines the topics pertaining to the
general discussion.
68
Study 1 General Discussion for Study 1
1. Descriptions of participants’ typical experiences
a. A participant’s experience with the concurrent think aloud
method
b. A participant’s experience with the retrospective and virtual
revisit website task
i. A participant’s experience with the retrospective
think aloud
ii. A participant’s experience with the virtual revisit
think aloud
2. Benefits and limitations
a. Concurrent think aloud method
b. Retrospective think aloud method
c. Virtual revisit think aloud method
3. Study 1 Limitations
4. Significance and Educational Implications
Figure 3.5. Topics pertaining to the general discussion of study 1.
A Participant’s Experience with the Concurrent Think Aloud Method
“I’m always [thinking] in my head, but I find when I’m talking I probably don’t do as
good of a job as when I’m just thinking…I’m maybe not absorbing as much information
either” (interview transcript, concurrent condition)
A participant in the concurrent condition receives the full set of instructions (the website
task and think aloud) and completes the practice task on a second educational website. The
participant immediately develops a sense of self-awareness, and feels awkward and slightly
69
stressed about the proceeding task of navigating the website while simultaneously thinking
aloud. The homepage of The Balanced Literacy Diet website appears on the laptop directly in
front of the participant. A digital recorder is activated, ready to capture the participants’
verbalizations as she navigates the website.
Initial verbalizations describe the setup of the homepage and include text, headings and
subheadings. The participant scans the text and reads words and phrases. She does this as she
quickly scrolls down the page. Then the participant notices the tabs at the top of the homepage.
She scrolls over the tabs before she selects a possible topic of interest: reading comprehension.
The reading comprehension page opens in a new window and the participant shares her
thoughts—she states that she seeks information related to her current grade.
The participant continues her navigation—she moves the computer mouse over the text
of the food group pages and sometimes highlights the text as she reads out headings,
subheadings, or the first sentence of a paragraph. At about the ten-minute mark, the participant
notices the embedded video on the right side of the screen and clicks on the video image. The
video is just over one minute in length; however, the participant stops it within the first 20
seconds. She remarks, slightly laughing to herself, that she is impatient and unable to watch the
entire video. Then the participant refers to her general website seeking behaviour—she usually
skims through websites until a concept catches her eye.
The navigational pattern continues over the course of the 20 minute task: the participant
opens pages of interest, skims the text, hovers over text and images, reads sections of text,
describes what she sees, and views snippets of video clips. Occasionally the participant states
her opinion about website features, like the quality of the photographs, and how the tools are
user friendly or cause confusion. Occasionally the participant states her understanding of a
concept. Most often the participant reads the text and describes the website features. The
70
participant rarely expands on her decisions or provides an explanation for her behaviour. The
participant’s navigation and think aloud end at the 20-minute mark.
A Participant’s Experience with the Retrospective and Virtual Revisit Website Task
A participant is given the first set of instructions: the website task instructions. The
screen capture program begins to record the computer screen and the participant begins her 20-
minute navigation on the homepage. First, she moves the computer mouse over the text and
over images, and slowly scrolls down the page. After approximately two minutes, the
participant notices the virtual tour monthly features on the left side of the screen. She selects the
first and second grade virtual tour. The virtual tour opens and the participant hovers over the
hot spots to view the titles of the video clips. Then, she uses the arrows to maneuver around the
virtual tour; occasionally she stops to view an image on the wall. The participant opens three
videos during her virtual classroom tour. She watches two in their entirety and stops the third
video as it approaches the end. She closes the virtual tour and navigates back to the homepage.
From here, the participant hovers over all of the tabs before she selects motivation for literacy
within the food groups tab. When this web page appears, the participant once again hovers over
text and images before she slowly scrolls down the screen. Then the participant selects an
embedded video. As the video plays, the participant maneuvers around this food group page;
she scrolls down again and opens a recipe page. She views the recipe while the video on the
food group page continues to play—this participant is multitasking. When the video ends the
participant opens another page from the food groups tab, writing conventions. The same pattern
of behaviours continues on this page and the remaining pages the participant visits: she slowly
scrolls down each page, views videos, and opens links to educational resources and photo
galleries.
71
A Participant’s Experience with the Retrospective Think Aloud
“To me I think [thinking aloud] is an amazing tool. If you’re not shy about things and
you just trust the process there’s just a lot of learning that can come out it” (interview
transcript, retrospective condition)
A participant is given the second set of instructions, the think aloud instruction, and the
audio recorder begins to record the participant’s verbalizations. The participant begins her think
aloud with confidence and certainty of her initial web-based actions. She immediately connects
her web-based behaviour to her current practice. Occasionally the participant states her opinion
and shares how she would like to implement certain aspects from the website into her practice.
At about two minutes into her think aloud, the participant pauses and asks, “What else did I
do?” She lists a number of actions, and then shares her overall sense of the website and her
general experience with online resources. The participant concludes her think aloud by saying:
“That’s all, I think.” The audio recorder is stopped at four minutes.
A Participant’s Experience with the Virtual Revisit Think Aloud
“It’s like you’ve got these two narratives: you’ve got this physical manifestation of what
you’ve done and you know what you were thinking…and the storytelling of what was
going on…It’s like this disembodiment, but as you’re doing the storytelling you’re
bringing them back together again” (interview transcript, virtual revisit condition)
A participant is given the second set of instructions: the think aloud instructions. The
screen recording of this participants’ website navigation begins to play on the laptop computer
directly in front of him. At the same time an audio recorder begins to capture the participant’s
verbalizations. As the participant views the screen recording of his navigation, he describes his
actions on the homepage and includes a reason for his decisions. As the participant views the
screen recording of his navigation he completed only moments ago, he views the cursor on a
72
link to the charitable organization associated with the development of the website. The
participant shares his reason for making this decision—he is interested in the credibility of the
site. After the participant spends time looking at this outside link, he views the screen recording
return to the homepage. The participant states that he was interested in learning more about the
framework of the site. This is the reason he provides for selecting the introductory page. As the
introductory page appears, the participant verbalizes his understanding of the website in relation
to his prior knowledge about the stages of literacy and literacy instruction. Then, the participant
reflects that at this point in his navigation he was ready to delve deeper into the site. The
participant says that he was seeking information related to his current practice. Specifically, he
expresses interest in implementing integrated art and writing activities with his students. A
pattern of verbalizations related to planning and professional goals, as well as to his students’
interests and needs continues. The participant often remarks on the ease of use of the website
tools and features as they appear and is likely to provide an explanation or reason for his
navigational decisions. At one point, the participant’s think aloud begins to veer off in an
unrelated direction, but he is immediately pulled back into his navigation as the screen changes
and a video appears. At the 20-minute mark the computer screen becomes blank and the
participant completes his final thought related to his final action.
Benefits and Limitations of Employing Each Type of Think Aloud Method
The previous section of the general discussion shared portraits of participants’
experience with each type of think aloud method. These descriptions made implicit references
to the benefits and limitations of employing each type of think aloud method in the context of a
professional development website. This section discusses the main benefits and limitations of
employing each type of think aloud method based on the above theoretical portraits and results
of study 1. Table 3.11 provides a summary of this discussion.
73
Concurrent think aloud. Findings from this study reveal one main benefit and three
limitations of employing the concurrent think aloud method in the context of a professional
development website. First, the concurrent think aloud generates direct data about the ongoing
cognitive processes that occur during task performance. Since the two activities, thinking aloud
and task performance occur simultaneously the verbalizations are valid forms of information—
the verbalizations contain direct data about participants’ thoughts in real time. The limitations
of the concurrent method, however, may outweigh this benefit. The first limitation of the
concurrent think aloud method is the fact that this think aloud produces fewer verbalizations
related to higher order cognitive processes. Secondly, the concurrent think aloud method results
in a high cognitive load on working memory. In addition, the concurrent think aloud may be an
uncomfortable procedure for participants.
Retrospective think aloud method. Findings from this study show three benefits and
two limitations of employing the retrospective think aloud method. First, the main benefit of
employing the retrospective procedure is that this method produces verbalizations related to
higher level cognitive processes (planning, connecting, and reflecting). Secondly, participants
are likely to demonstrate a high level of engagement with the website task. In addition,
participants will most likely be comfortable during the process of thinking aloud. The
limitations of employing the retrospective think aloud method, however, are significant.
Participants will omit most of their online actions and navigational decisions. Furthermore,
participants will most likely have difficulty recalling their intermediate web-based actions and
reasons for these decisions.
Virtual revisit think aloud method. Findings from this study show four benefits and
one limitation of employing the virtual revisit think aloud method. First, the main benefit of
employing the virtual revisit procedure is that this method produces verbalizations related to
74
higher level cognitive processes (planning, reasoning, and reflecting). Secondly, participants
can rely on a visual aid to help them recall their navigational decisions and why they made
them. The third benefit of employing the virtual revisit think aloud method in the context of a
professional development website is that participants are likely to demonstrate a high level of
engagement with the website task. Finally, participants will likely feel comfortable with the
process of thinking aloud. One limitation of the virtual revisit think aloud is the time required to
complete both the website task and think aloud task. In the current study, the total time for each
participant in the virtual revisit condition was 40 minutes. For some researchers, this may be
costly. Moreover, the time required to employ the virtual revisit think aloud may limit
participant involvement.
Table 3.11
Benefits and Limitations of Employing Each Type of Think Aloud Method
Concurrent Think Aloud
Method
Retrospective Think
Aloud Method
Virtual Revisit Think
Aloud Method
Benefits Generates direct data
about the ongoing
cognitive processes
that occur during task
performance
Produces
verbalizations related
to higher level
cognitive processes
(planning,
connecting,
reflecting)
Participants likely
will feel comfortable
with thinking aloud
High task
engagement
Produces
verbalizations related
to higher level
cognitive processes
(planning, reasoning,
reflecting)
Participants can rely
on a visual aid to
recall decisions
Participants likely
will feel comfortable
with thinking aloud
High task
engagement
Limitations Produces
verbalizations related
to lower level
cognitive processes
(descriptive, reading)
Errors of omission
Difficulty recalling
intermediate actions
Time consuming to
conduct
75
High cognitive load
on working memory
Participants may feel
uncomfortable with
thinking aloud
Collectively, the findings of study 1 provide support for the virtual revisit think aloud as
an effective method for examining elementary teachers’ cognitive processes as they use a
professional development website. Results indicate that the virtual revisit can avoid the
limitations of the concurrent and retrospective procedures and provide thorough and descriptive
thought units and insights into how teachers use and learn from a professional development
website.
Study 1 Limitations
There were four main limitations to this study: (1) a wide range of variability in
participants within the conditions, (2) a small sample size, (3) factors that may have caused
reactivity, and (4) the use of one professional development website. First, some of the variables
had a large standard deviation indicating that the data points were distant from the mean or more
spread from the mean. While the effect sizes for the significant findings were fairly strong, the
wide range of variability needs to be taken into consideration when interpreting the findings.
Individual differences may account for the wide range of variability, including background
knowledge, motivating factors, and comfort level with speaking in front of adults.
Secondly, the small sample of teachers assigned to each condition indicates a decrease in
power of the statistical tests and therefore a cautious interpretation of the findings is
recommended (Field, 2009). While future research could benefit from a larger sample size of
20 or more teachers in each condition, the effort and cost in recruiting practicing teachers and
conducting the think aloud tasks are noted. Recruiting 45 practicing teachers to volunteer to
76
participate in the current study was a challenge and alternative modes of recruitment, including
word of mouth, were employed. Gift cards valued at $20 were given to teachers who
participated in this study; however a greater incentive to participate might result in a larger
sample size.
Thirdly, factors may have caused reactivity during the think aloud procedure. As
previously mentioned, reactivity occurs when task performance is altered as a result of an
awareness of the study task. Reactivity may have occurred as a result of participants’ awareness
that they were completing a task in the presence of the primary investigator. A “motivational
shift” in which the participants anticipate exposure of their think aloud protocol may have
occurred when the participants were informed of the think aloud procedure (Russo, 1989).
Another factor that may have caused reactivity is hearing one’s own voice. As previously
mentioned, the additional aural stimulation may have interfered with the concurrent navigation
(Russo, 1989). In general, reactivity was reduced as much as possible during the one-on-one
meeting by: staying neutral during the task and think aloud, keeping verbal and nonverbal cues
to a minimum, and providing participants in the retrospective and virtual revisit conditions with
the second part of the instructions after they had completed the website task. However, the
factors described above should be considered when interpreting the findings.
Finally, the current study was context-specific and used one professional development
website. Future research comparing the three think aloud conditions should be conducted with
additional websites and online resources. Conducting similar studies with alternative
professional development websites will enhance the credibility and transferability of the results.
Significance and Educational Implications
The significance of this research is that it compares the effectiveness of two traditional
think aloud methods—concurrent and retrospective—with a think aloud method combining a
77
retrospective procedure with screen capture technology (the virtual revisit). While think aloud
research is extensive, studies that compare the effectiveness of different think aloud
methodologies for understanding teachers’ cognitive processes as they navigate online resources
are scarce. Based on the findings of this study, the virtual revisit think aloud is an effective
method for examining elementary teachers’ cognitive processes as they use a professional
development website. The virtual revisit think aloud method produces rich and thorough
verbalizations related to teacher planning; the virtual revisit allows participants to incorporate
reasons behind the decision-making process; and the virtual revisit provides opportunities for
reflection. In addition, because participants think aloud immediately following the website task,
participants engage with the website as they might normally do when seeking information
online related to their practice. Participants will also likely feel comfortable with the think
aloud procedure.
Another significance of the current study is that the participants were practicing teachers
with between one and over 25 years of classroom experience. Much of the research examining
teachers’ beliefs and attitudes towards learning in online environments has studied preservice
teachers. Studying the cognitive processes of practicing teachers has the potential to contribute
to the understanding of teacher professional development and teacher cognition.
Furthermore, the virtual revisit think aloud method also has potential that transcends the
specific domain in which it is applied in the present study. Virtual revisits with think aloud
could be applied to examine cognitive processes of participants in research involving online
learning and website use in many domains in education. If researchers provide feedback to
educational website developers based on the virtual revisit methodology, improvements can be
made to the design and content of their sites.
78
Additionally, the virtual revisit think aloud method could be used as a tool for teacher
professional learning. The benefits of commenting on a screen recording of navigational
behviours in the context of professional learning may include encoding information and long
term memory storage. Many of the participants in the virtual revisit condition discussed their
own learning and self-reflection during their think aloud and how the process of viewing their
online behaviours while simultaneously thinking aloud had an impact on their reflective process
of teaching and learning. This process produced connections to prior knowledge allowing
teachers to encode information to long term memory.
Conclusion
Teachers learn in many different aspects of practice, including their classrooms, their
school communities, professional development courses, and online environments (Borko, 2004).
To understand teacher learning, it must be studied within these multiple contexts and it must be
studied with an effective methodology that provides rich and thorough data about the reasoning
process. Gaining greater insights into teachers’ cognitive processes as they navigate online
environments can lead to the reconsideration of the design of online learning environments so
that they “are more conducive to informal learning…so that they further develop the ability of
professionals to solve problems and learn independently” (Lohman, 2006, p. 144). Based on the
findings of study 1, the virtual revisit think aloud methodology provides a comprehensive
picture of elementary teachers’ cognitive processes as they use and learn from a multimedia
professional development website.
79
CHAPTER 4
Understanding Teachers’ Learning Experiences as They Use a Professional Development
Website: A Qualitative Study
The aim of study 2 is to contribute to the field of teacher cognition and online learning
by exploring how elementary teachers use and learn from an online professional resource.
Study 2 utilizes the virtual revisit think aloud method and two additional data sources (a
demographic questionnaire and a semi-structured interview) to examine elementary teachers’
learning experiences as they navigate a multimedia professional development website. As
previously described, the virtual revisit think aloud is a method that combines a retrospective
think aloud and screen capture technology. Results from study 1 suggest that the virtual revisit
think aloud method generates comprehensive information about teachers’ cognitive processes as
they make online choices. In the current study, the virtual revisit think aloud offers insights into
elementary teachers’ learning experiences as they use The Balanced Literacy Diet website.
Chapter 4 presents study 2 in six major sections: (1) literature review, (2) study purpose
and research questions, (3) methods, (4) findings and discussion, (5) general discussion, and (6)
limitations, implications, and future directions. First, a discussion of relevant literature in the
areas of adult learning, and formal and informal professional development sets the foundation
for study 2. Next, this chapter presents the study purpose and research questions. The third
major section of this chapter is the methods section which begins with participant demographics
relevant to study 2. The methods section continues with an overview of the research design,
data analysis, and the techniques used for establishing trustworthiness of the qualitative
findings. The fourth major section of this chapter presents the findings and discussion, and
highlights nine major themes related to the research questions. A general discussion follows the
nine themes. Finally, this chapter concludes with a discussion of the limitations, significance
and implications, and future directions.
80
The following review of the literature includes two main topics relevant to study 2: (1)
models of adult learning, and (2) formal and informal professional development. The first topic
highlights three models of adult learning. The first model is Malcolm Knowles’ theory of adult
learning (1980). Knowles was a theorist of adult education who popularized a theory comprised
of several underlying characteristics of the adult learner. Because study 2 investigates teachers’
learning experiences as they navigate a professional development website, Knowles’ theory
offers the current study a foundational view of the nature of teachers as learners.
The second model of adult learning is self-directed learning. As one of the primary
characteristics in Knowles’ theory of adult learning and particularly relevant to the use of
informal online environments, self-directed learning warrants further attention. The model of
self-directed learning incorporates three primary principles: (1) a self-initiated process of
learning, (2) personal autonomy, and (3) greater learner control (Caffarella, 1993). These
principles relate to teacher motivation for learning and to the context of informal online
professional development.
The third model presented in this chapter is Shulman and Shulman’s (2004) model of
teacher learning. Shulman and Shulman’s model emphasizes five aspects related to teacher
learning: vision, motivation, understanding, practice, and reflection. These aspects comprise a
process of teacher learning which has applications for online professional development
activities.
The second section of the literature review presents two types of professional
development: formal and informal. While the emphasis of the current study is informal
professional development in online environments, this chapter briefly discusses formal
professional development. Formal and informal professional development activities are
81
discussed with respect to elements of teacher learning in the context of online professional
development.
Theoretical Models for Understanding Teacher Learning
Knowles’ Theory of Adult Learning
Knowles’ theory of adult learning is based on the concept andragogy: “the art and
science of helping adults learn” (Knowles, 1977, p. 211). Grounded in the concept that adults
learn differently from children, Knowles popularized the term andragogy in the late 1960s when
he developed a theory about adult learning. Five underlying characteristics of the adult learner
can be drawn from Knowles’ theory of adult learning: (1) self-directedness, (2) the use of
experience as a source of information, (3) immediate application of knowledge, (4) need to
know, and (5) intrinsic motivation (Knowles, 1980; Merriam, 2001). What follows is a
discussion of each characteristic in relation to teacher professional development.
Self-directed learners are free from external control and constraint. Teachers are self-
directed learners when they choose educational topics that directly relate to their individual
practice and classroom context (Trotter, 2006). Choice in professional development activities is
important for two reasons: it reinforces teachers’ status as professional, and it increases the
chances that activities will apply to an individual teacher’s learning needs and professional
interests (Brockett & Hiemstra, 1985; Haymore Sandholtz, 2002; Knowles, 1980; Merriam,
2001). Teachers who pursue meaningful topics are active participants during the learning
process—they independently make choices that relate to their distinct learning pursuits
(Cercone, 2008; Knowles, 1980).
Professional experience is a source of information for teachers and can be highly
valuable during the process of learning. That is, when teachers create connections between new
incoming information and past experiences, the new information is more likely to be retained
82
for future use. Teachers can use their experience to verify the relevance of newly learned
material and to envision possible future implications (Cercone, 2008).
Teachers value material that they can easily and immediately transfer to their practice.
As teachers construct new ideas from professional development resources, they might envision
how the new information relates to their classroom and ultimately how it can contribute to
meaningful student learning experiences. Initially, teachers might experiment with their newly
formed knowledge, often tailoring new information to their unique environment. Teachers are
also more likely to commit to new instructional methods on a long-term basis when these
methods are successful with their students (Guskey, 2002).
Teachers can be considered goal-oriented and often need to know the reason behind the
learning. Specifically, teachers value the application of theory to practice. When teachers can
visualize the practical application of a theory, they may be more likely to embed theory into
their daily instructional activities. In addition, teachers may be more likely to invest in material
that they deem meaningful to their classroom environment and professional goals.
The application of newly acquired knowledge is more likely to occur when teachers are
intrinsically motivated—when engagement in learning is inherent without the mediating effects
of external rewards (Gorozidis & Papaioannour, 2014). That is, an internal drive to learn and
willingness to engage in professional development activities increases when the demands of
external factors (for example, mandatory workshops) are absent. In addition, engagement
increases when the material is relevant and personally meaningful to the learner (Chametzky,
2014).
Taken together, the above characteristics of the adult learner are relevant to teacher
professional development (Knowles, 1975; Knowles et al., 2011; Merriam, 2001; Schellenbach-
Zell & Grasel, 2010; Trotter, 2006). Teachers may be more willing to engage in professional
83
development opportunities that they perceive as meaningful; they may be more likely to gain
greater subject knowledge and transform their instructional practices when professional
development fosters self-directedness, connects to their experiences, and is easily and
immediately transferrable. In addition, when professional development is internally driven,
positive outcomes in teacher learning and in the application of theory to practice are more likely
to occur. In sum, teachers actively construct knowledge through “an interactive process of
interpretation, integration, and transformation of [their] experiential world (Pratt, 1993, p. 17).
For the purposes of study 2, Knowles’ theory provides a foundational view of the nature
of adult learning and can be applied to teacher learning in the context of online professional
development; however three criticisms of his theory require attention. The first criticism
pertains to the theory’s focus—the characteristics focus on the individual rather than on the
learning process (Hartree, 1984; Rachel, 2002). While an understanding of the characteristics of
adult learners can contribute to instructional designs and methods that best meet the needs of
adult learners, Knowles theory omits elements involved during the learning process; for
example: stages of learning, cognitive strategies, and potential learning outcomes. In addition,
the characteristics do not necessarily occur in a particular order; for instance, self-directedness is
not a requirement for using experience as a source of information.
A second criticism of Knowles’ theory relates to a developmental continuum; the
characteristics that describe the adult learner are not necessarily true for all adults, and can also
be true for children. Regardless of age, learners should be viewed along a continuum of each of
the characteristics, and individual differences affect how learners approach different situations
and learning activities. For instance, experienced teachers might feel very comfortable self-
directing their learning, whereas novice teachers might initially require guidance from an
administrator.
84
Finally, Knowles’ theory has been criticized for the limitations in the validity of the
theory—few researchers have performed empirical studies that directly investigate the theory
(Pratt, 1993; Rachel, 2002). Studies that do investigate Knowles’ theory show “considerable
variation in the implementation of andragogical methodologies” (Rachel, 2002, p. 217). That is,
Knowles’ theory lacks specific procedures for adult educators to undertake to best meet the
needs of adult learners (Taylor & Kroth, 2009). This lack of empirical evidence suggests
Knowles’ theory can only guide our understanding of how adults learn.
While the above criticisms are worth mentioning, Knowles’ characteristics of the adult
learner can be considered during the design and development of professional development
activities. Recognizing the characteristics of adult learners can foster a learner-centred approach
to teacher professional development (Cercone, 2008, p. 141).
Self-Directed Learning Theory
Self-directed learning is a well-established notion (Brockett & Hiemstra, 1985; Knowles,
1975) and stems from an individual’s desire to understand a phenomenon, an incident, or a
concept (Ponti, 2014). As previously mentioned, the model of self-directed learning
incorporates three principles: a self-initiated process of learning, personal autonomy, and greater
learner control (Caffarella, 1993). Each of these concepts is discussed with respect to teacher
professional development.
As a self-initiated process of learning, self-directed learning emphasizes “the ability of
individuals to plan and manage their own learning” (Caffarella, 1993, p. 25). The expectation of
self-directed learning is that the individual learner assumes “primary responsibility for their own
learning” depending on their unique needs and individual goals (Caffarella, 1993, p. 26). When
given the opportunity, teachers might self-direct their learning by first identifying their needs.
Then, teachers may decide which professional development approach will best meet their needs.
85
Approaches to professional development might include online professional development
activities, such as distance education courses or professional development websites. Teachers
might use Internet technologies and web-based resources to construct knowledge and identify
instructional materials relevant to their teaching context and professional goals.
A central trait of a self-directed learner is autonomy. Personal autonomy implies
“independence, freedom of choice, and rational reflection” (Loyens, Magda, & Rikers, 2008, p.
414); teachers may independently make choices according to their needs and interests. In
addition, personal autonomy is positively related to personal values and beliefs (Caffarella,
1993). A stronger sense of personal values and beliefs may contribute to a “solid foundation for
conceiving goals and plans” (Caffarella, 1993, p. 29). Teachers who conceive professional
goals and instructional plans that are personally valuable may be more likely to integrate these
ideas into their practice.
In addition, autonomy is individualistic (Candy, 1988) and autonomous behaviour ranges
according to several factors: the learners’ “level of technical skills, their familiarity with the
subject matter, their sense of personal competence as learners, and the context of the learning
event” (Caffarella, 1993, p. 30). These factors may affect how teachers approach professional
development, particularly when accessing web-based technologies. That is, experience with
technology may contribute to increases in autonomy during online learning activities.
Greater learner control implies that the learner is the one who considers the content,
approach, and value to the learning experience (Caffarella, 1993). With greater learner control
individual needs are more likely to be met in teachers’ quest for pedagogical knowledge and
instructional materials. Moreover, greater learning control may be linked to higher levels of
motivation for learning.
86
Taken together, professional development activities that foster a self-initiated process of
learning, personal autonomy, and greater learner control may contribute to higher levels of
motivation for learning and the application of pedagogical knowledge to the classroom.
Informal online learning environments lend themselves to a “self-directed learning experience”
(Song & Hill, 2007, p. 35). Teachers who initiate online professional development activities
with a sense of self-directedness, autonomy, and learner control may benefit from the
experience—they become motivated to learn and to implement best practices in their classroom.
Shulman and Shulman’s Model of Teacher Learning
The third theoretical model presented in this chapter for understanding how teachers
learn is a model proposed by Shulman and Shulman (2004). Shulman and Shulman highlight
five aspects of teacher learning: vision, motivation, understanding, practice, and reflection
(2004). Each aspect is discussed next.
Shulman and Shulman’s model of teacher learning begins with vision. As teachers enter
the classroom for the first time in their career, at the beginning of a new school year, or each day
they have a vision about their students’ learning, classroom environment, and instructional
methods. This vision guides teachers in their practice and can influence how they approach
professional development activities.
Along with vision, a teacher’s motivation and willingness to learn contributes to positive
outcomes, including the construction of new pedagogical knowledge and the implementation of
meaningful activities in the classroom. As previously described, teachers tend to be motivated
to learn; particularly in the absence of external demands as well as when material is relevant and
personally meaningful (Chametzky, 2014).
Through experience teachers develop an understanding of the elements that form the
knowledge-base of teaching; for example, pedagogical content knowledge, curriculum and
87
subject knowledge, and a developmental perspective of the learner. These understandings
contribute to their daily practice.
According to Shulman and Shulman (2004), a teacher’s practice is set upon a foundation
of vision, motivation, and understanding. When this foundation has been established, a teacher
can more easily consider a range of ongoing factors related to their practice; such as, the design
and adaptation of curriculum, classroom management, formal and informal assessments, and
student diversity (Shulman & Shulman, 2004). A teacher’s practice develops and has the
potential to transform through ongoing reflection. That is, when teachers are conscious of their
own understandings, performances, and dispositions, their capacity to learn is enhanced
(Shulman & Shulman, 2004).
Taken together, Shulman and Shulman’s model of teacher learning highlights five stages
of the learning process. Each stage can be applied to the context of online professional
development. That is, as teachers use the Internet for their professional learning they initiate
their search for pedagogical information with a vision about their students, teaching context, and
instructional methods. This vision is enhanced by a teacher’s willingness to learn—an internal
motivation. The construction of knowledge and development of deep understandings via online
resources contributes to meaningful practice. As teachers reflect on their learning and practice,
new visions arise and the cycle of professional learning continues.
In sum the three theoretical models discussed above highlight teacher learning as an
integrated set of various factors. Given that the current study examines the processes of teacher
learning in the context of a professional development website, the next section discusses formal
and informal online learning environments.
88
Formal and Informal Professional Development
Learning is an activity that occurs in a range of contexts. In any learning context—
formal, informal, or along a continuum—there is the potential for knowledge growth, changes in
beliefs, and transformations in practice (Bakkenes et al., 2010). Formal professional
development can introduce teachers to educational innovations and evidence-based methods of
teaching in content areas (Jurasaite-Harbison & Rex, 2010), and includes, but is not limited to,
face-to-face workshops and distance education courses. These types of professional
development activities are often structured, and usually guided by an instructor or set of
expectations. In contrast, informal professional development is usually open-ended, flexible,
and self-directed. Informal professional development activities are likely to coincide with a
teacher’s interests and often consider individual processes, real-life activities, and contextual
factors (Callanan, Cervantes, & Loomis, 2011). Teachers choose informal approaches,
including conversations between colleagues and online resources, to informally construct
knowledge and improve their instructional practice. Informal professional development
provides teachers with innovative strategies and research-based methods of instruction in a way
that directly relates to goals and interests of individual teachers.
Within the domain of informal professional development is independent teacher
learning: “learning activities that teachers engage in on their own initiative and accord, and
which possess no connection to their organization” (Jones & Dexter, 2014, p. 371). Independent
learning aligns with the characteristics of adult learning, as described above. For instance, a
central component of independent learning is self-direction, in which teachers independently
choose to pursue meaningful topics that directly relate to their personal interests and individual
needs. Independent learning contexts also promote the use of experience as a source of
information and allow meaningful connections between new incoming information and past
89
experience. Moreover, independent learning contexts are not restricted by time and location;
they are accessible, and convenient. This anytime-anywhere approach to professional
development promotes the immediate application of teachers’ newly formed knowledge, and the
reason behind the learning is often known. Furthermore, independent learning is often driven by
an internal motivation to learn—adults have a basic need for autonomy (Deci & Ryan, 2008). In
sum, independent learning contexts can consider the characteristics of the adult learner. In
addition, independent learning contexts lend themselves to a self-directed learning experience—
one that fosters autonomy and greater learner control.
A recent study compared formal, informal, and independent learning (Jones & Dexter,
2014). Findings indicate that while teachers reported general satisfaction with formal learning
opportunities, they identified constraints, lack of choice, and ongoing support as limitations of
these formal activities (Jones & Dexter, 2014). The findings also specified informal and
independent learning (i.e., Internet resources) as key components of teachers’ overall
professional learning experiences and independent learning as a primary activity crucial to
teacher learning (Jones & Dexter, 2014). The study also found that informal learning activities
were often initiated by teachers; participants in the study self-directed their own learning.
In another study that examined factors that influence teachers’ engagement in informal
learning activities, personal characteristics appeared to enhance teachers’ motivation to engage
in informal learning (Lohman, 2006). These personal characteristics include initiative, self-
efficacy, love of learning, interest in the profession, commitment to professional development, a
nurturing personality, and an outgoing personality. Findings also identified three factors that
inhibit teacher professional development: a lack of time, a lack of proximity to colleagues’ work
areas, and insufficient funds. These findings suggest that accessible online professional
90
development can offer an alternative to the more traditional and formal professional
development activities.
Informal online learning environments have become increasingly popular to teacher
professional development and the outcomes of informal online learning are positive: for
instance, informal online learning fosters self-directedness, topics of interest are easily
accessible, and there is a sense of control and choice in the learning. However, there are also
challenges to informal and self-directed online professional development. One challenge of
informal online learning is “that it is up to each teacher to interpret the professional
development content and translate it into specific action in their classroom” (Fishman et al.,
2013, p. 427). Teachers may have access to a curriculum leader or administrator for
clarification and guidance; however, the teacher independently accesses and interprets given
information. Another challenge of informal online learning is the lack of hands-on experience
and collaboration between colleagues. It is well-known that teachers come together to share
their experiences and guide each other in their learning—“teachers can be valuable resources for
each other’s learning in professional development” (Fishman et al., 2013, p. 427). During self-
directed online learning, collaboration may be limited. One way to overcome this limitation is
to encourage teachers to share learning resources during professional learning communities and
staff development sessions, especially resources teachers deem as credible and highly valuable.
The current study acknowledges collaboration as an important aspect to teacher
professional development and considers informal online learning as just one of the many facets
of professional development. Teacher professional development is complex and multifaceted,
and as teachers continue to use the Internet for their professional learning, it is necessary to
examine the processes and experiences involved during informal online learning. Informal
learning experiences have been shown to have a positive effect on, and even change, teachers’
91
beliefs, acquired knowledge and pedagogical approaches (Cervetti et al., 2012). Therefore, the
aim of study 2 is to contribute to the scholarship of teacher learning and professional
development by examining how teachers independently use and learn from one informal
professional development website.
Study Purpose and Research Questions
The purpose of this qualitative study is to gain an in-depth understanding of elementary
teachers’ learning experiences as they navigate a multimedia professional development website.
Despite extensive research on teacher learning and professional development, there are few in-
depth studies that examine elementary teachers’ learning experiences as they utilize informal
online resources. Specifically, little to no research has examined how teachers use and learn
from a multimedia professional development website, the cognitive processes and strategies
involved, and potential outcomes related to teacher learning and professional goals.
In-depth analyses of elementary teachers’ online learning experiences can offer a deeper
understanding of teacher learning within the complex field of continued professional
development. In addition, research in the area of informal online learning can offer website
developers and professional development administrators with suggestions pertaining to the
development and refinement of professional development websites. The following research
questions guided this study:
1. How do elementary teachers use a multimedia literacy professional development
website?
2. What factors influence how teachers use a literacy professional development
website?
3. What cognitive processes and strategies do teachers employ as they navigate a
literacy professional development website?
92
4. What are some of the potential outcomes of using and learning from a literacy
professional development website?
Methods
The methods section of this chapter describes the participants, research design process,
data analysis, and techniques for establishing trustworthiness of the findings. The methods
section begins with a description of the demographic characteristics of the participants in study
2 (the participants from the virtual revisit condition). Then, the methods section of this chapter
presents the research design and the overall process for study 2. Next, the data analysis
highlights three phases of coding: open coding, axial coding, and selective coding. The methods
section of study 2 ends with a description of the strategies and techniques used for establishing
trustworthiness of the qualitative findings.
Participants
Study 2 analyzes a sub-sample of participant data from study 1. Specifically, study 2
further examines data from the virtual revisit condition. The results of the demographic
questionnaire revealed that, in the virtual revisit sample, (N = 15) male participants (n = 3, 20%)
represented a smaller proportion than female participants (n = 12, 80%). The majority of
participants from the virtual revisit sample were between the age of 30 and 39 (n = 8, 54%) and
most participants had between five and nine years of teaching experience (n = 5, 33%).
Furthermore, most of the participants in the virtual revisit sample taught at the primary level (n
= 7, 47%). In addition, 13 participants (87%) reported feeling very comfortable using the
Internet for professional purposes and two participants (13%) reported feeling somewhat
comfortable using the Internet for professional purposes. The majority of participants in the
virtual revisit condition reported using the Internet for professional purposes at least once a day
93
(n = 7, 47%). Table 4.1 provides details concerning the demographic characteristics of the
participants in study 2.
Table 4.1 Demographic Characteristics for Study 2
Characteristic Frequency and Proportion
n (%)
Gender
Male 3 (20%)
Female 12 (80%)
Age
25-29 3 (20%)
30-34 4 (27%)
35-39 4 (27%)
40-44 2 (13%)
45-49 1 (6%)
50-54 0
55+ 1 (6%)
Years Teaching
2-4 years 4 (27%)
5-9 years 5 (33%)
10-14 years 4 (27%)
15-19 years 1 (6%)
20-24 years 1 (6%)
25+ years 0
Current Grade
Kindergarten 0
Primary 7 (47%)
Junior 6 (40%)
Primary/Junior 2 (13%)
Comfort Using Internet for
Professional Purposes
Very 13 (87%)
Somewhat 2 (13%)
Not very 0
Frequency of Internet Use
for Professional Purposes
> once/day 6 (40%)
Once/day 7 (47%)
Once/week 1 (6%)
Once/month 1 (6%)
94
Research Design
Study 2 employs qualitative methods to gain an in-depth understanding of elementary
teachers’ learning experiences as they utilize a multimedia professional development website.
Qualitative methods follow the post-positivist paradigm, which uses a naturalistic approach to
understand phenomena in a context-specific setting (Golafshani, 2003). In contrast with the
positivist paradigm, which seeks to explain, predict, and generalize to other contexts, the
primary aim of qualitative research methods is to describe, understand, and interpret phenomena
using an inductive approach to data analysis (Lincoln & Guba, 1985). An inductive approach to
analysis provides insights into how individuals experience a particular phenomenon and the
processes involved in a particular situation (Rowan & Huston, 1997).
There are several approaches to qualitative research (e.g., case study, phenomenology,
and ethnography) and choosing an approach depends on the study purpose and context. The
current study employs a grounded theory approach (Glaser & Strauss, 1967). The goal of
grounded theory is to generate a general explanation or theory of a process, action, or interaction
(Creswell, 2007). This general explanation or theory “is derived from data and then illustrated
by characteristic examples of data” (Glaser & Strauss, 1967, p. 5). Grounded theory was
deemed an appropriate qualitative approach for the current study for two reasons: (1) there is a
limited amount of research that examines in-depth how elementary teachers use and learn from
multimedia online learning environments, and (2) a theoretical model for elementary teachers’
online learning experiences as they use a professional development website can guide
researchers in the development and refinement of professional development websites.
The grounded theory research process for study 2 begins with three different data
sources from the virtual revisit condition: virtual revisit think aloud, interview, and demographic
data. Based on these data sources, the data set for study 2 includes three components: think
95
aloud transcripts, interview transcripts, and demographic questionnaire responses. These
components were analyzed during three phases of coding. The first and second phases of
coding, open coding and axial coding, involved an examination of the think aloud transcripts, as
well as memo writing and member checking. The interview transcripts and demographic
questionnaire responses were analyzed during the latter stage of axial coding. During the third
phase of coding, selective coding, the data were re-examined and a theoretical model was
developed.
The coding process led to nine major themes. These nine themes were organized into a
theoretical model, according to the grounded theory framework, and are discussed in the
findings and discussion section of this chapter. Figure 4.1 outlines the overall process for study
2. The following data analysis section describes the three phases of coding, memo writing, and
the member checking process.
Data Collection
Date sources:
demographic questionnaire
think aloud
semi-structured interview
Study Two Data Set
Three components:
15 think aloud transcripts
15 interview transcripts
15 demographic questionnaires
Phases of Coding and Procedures
1. Open coding:
Think aloud transcripts
Memo writing
2. Axial coding:
Think aloud transcripts
Interview transcripts
Demographic data
Member checking
96
Memo writing
3. Selective coding:
Think aloud transcripts
Interview transcripts
Demographic data
Nine Major Themes
1. Perception of professional learning
2. Focusing on student needs and
instructional goals
3. Individual differences
4. Navigating a professional
development website
5. Evaluating information
6. Encoding information for future
retrieval
7. Reflecting on learning
8. Continued professional learning
9. Intentions for practice
Theoretical Model Based on the
Grounded Theory Framework
Figure 4.1. Outline of overall process for study 2
Data Analysis
Data analysis in qualitative research involves reducing the data into themes as a result of
repeated coding, comparisons, and categorizations (Creswell, 2007). As mentioned above,
study 2 follows a grounded theory approach to data analysis and employs three phases of
coding: open coding, axial coding, and selective coding. These phases of coding involved data
categorization, comparisons, synthesis, and refinement, and led to a theoretical model for
elementary teachers’ online learning experiences as they used a professional development
website. As Charmaz (2014) states, “coding is the pivotal link between collecting data and
developing an emergent theory to explain these data…codes are created by defining what we see
in the data” (p. 46). The remainder of the data analysis section describes each phase of coding.
97
Open coding. In this study, open coding followed three stages. First, the think aloud
transcripts were read reflectively to gain a general sense of the participants’ thought processes.
Secondly, phrases within the transcripts were unitized based on meaningful pieces of
information that were “interpretable in the absence of any additional information” (Lincoln &
Guba, 1985, p. 345). The units consisted of simple sentences and longer phrases as opposed to a
specified number of words or lines within the transcripts. As the transcripts were unitized,
initial codes were generated based on participants’ actions and processes (Charmaz, 2014). For
example, the following unit was coded as tailoring information to teaching practice: “I was
reading the quick tips, and it seems easy enough that you can start a writing centre in any
classroom. You can tailor it very easily to the needs of your students.” The third stage of open
coding involved a process of constant comparison (Glaser & Strauss, 1967). As units were
coded, they were compared for similarities and differences. Units that were found to be similar
were grouped together. This resulted in an initial list of categories.
The initial list was extensive and included 77 categories (see Appendix C-1); so there
was an attempt to refine the list by, again, looking for instances of similarity. For example, the
units coded as connecting website content to future planning and classroom set up were found
to be similar to the units coded as constructing new ideas for future practice; therefore these two
categories were grouped together as knowledge construction. This process of comparing and
synthesizing categories began the next phase of coding and resulted in 18 main categories.
Memos were also produced during open coding. Memo writing provided a record of
initial thoughts and consisted of ideas related to possible connections between categories and the
grounded theory framework. As an example, the following memo describes initial thoughts
about the core phenomenon, navigating a professional development website, and the strategy,
encoding information for future retrieval:
98
As I’m beginning to read through the transcripts, it appears that participants reflect on
how they take in as much information as possible: they’re noticing images and materials
on the walls of classrooms, details behind a teacher who is speaking in a video, the
spatial organization of the classroom in relation to their own classroom and practice.
Participants attempt to absorb and remember information by encoding: they’re
connecting, taking notes, and planning.
Memo writing continued for the remaining two phases. Appendix C-2 presents a sample of
memos written during the analysis.
Axial coding. The second phase of analysis, axial coding, involved a review of the units
and categories. This review helped to establish connections between the categories and
subcategories and resulted in a refined list of categories. For example, the categories, current
grade and classroom context, narrowing down the focus, and reflecting on student needs and
goals, were grouped together under the theme, focusing on student needs and goals. This phase
of analysis identified seven preliminary themes:
(1) Focusing on student needs and goals;
(2) Evaluating information;
(3) Encoding information for future retrieval;
(4) Intentions for practice and professional learning goals;
(5) Reflecting on their learning and use of website;
(6) Valuing best practices in literacy and teaching;
(7) Feeling motivated for learning and applying new knowledge
To test the validity of these seven preliminary themes, member checking occurred next.
Member checking is a valid approach to establish the meaningfulness of the initial findings and
involves taking the ideas back to the research participants for their confirmation, elaboration, or
99
disagreement (Charmaz, 2014; Lincoln & Guba, 1985). Participants play a significant role in
contributing to the credibility of the findings; therefore, the participants (N= 15) were contacted
through email and asked if they were interested in providing feedback on the seven preliminary
themes. Specifically, participants received the following email script:
I am currently working on my analysis and as part of this process I am interested in
participant feedback. It will only take a few minutes of your time and your feedback
will be greatly appreciated. If you are interested in being involved I will share (email)
my preliminary findings with you and ask you to comment (as little or as much as
you would like) on seven preliminary themes. I will also email you the transcript of
your think aloud so that if you would like to, you can look over what you said during the
think aloud. Only I will have access to your comments and your name will not be used
in any paper or report.
Ten participants replied and were willing to provide feedback on the preliminary themes. These
participants were asked to review and comment on the preliminary themes in a follow-up email.
Appendix C-3 presents the themes and short descriptions sent to the participants. In general,
participants confirmed and elaborated on the seven themes: all of the participants involved in the
member check agreed with the themes. Appendix C-4 presents participants’ ‘member check’
comments.
Member checking contributed to the analysis in two ways: (1) by verifying the major
themes, and (2) by refining the major themes. First, the member check verified the validity of
the preliminary themes, as in the following example. In response to the theme, focusing on
student needs and goals, one participant stated, “I would definitely agree with this—when I
search, the most important thing I would initially be thinking about is: Is this relevant and
applicable to my students (age/grade/level) and will they benefit from it?” Another participant
100
made a similar connection: “This makes sense to me, as a teacher’s first goal is to support their
students. It’s the first check of how I evaluate any and all new resources.”
Member checking also aided in the refinement of themes. For example, the preliminary
theme, feeling motivated for learning and applying new knowledge, was refined into two
subthemes of a potential outcome, intentions for practice.
As a result of axial coding and member checking, nine major themes emerged. These
major themes were used to code the interview and questionnaire data. The purpose of analyzing
the interview and questionnaire data was to determine the consistency of the nine themes across
data sources. Appendix C-5 presents a distribution chart showing the frequency of themes
across data sources and across participants. The following section describes the last phase of the
grounded theory analysis, selective coding.
Selective coding. Selective coding is also known as theoretical coding. This process
involves selecting the core category and specifying relationships between the core category and
other categories to develop a theoretical model (Charmaz, 2014). In the current study, the
process involved organizing the nine major themes into a model based on four components of
the grounded theory framework:
(1) Conditions that affect the core category;
(2) The core category also known as the central phenomenon;
(3) Strategies as a result of the core category; and
(4) Potential outcomes related from the core category
The central phenomenon, navigating a professional development literacy website, was deemed
the core category because of the following three reasons, as outlined by Corbin and Strauss
(2008):
(1) the central phenomenon represents the main theme of the research;
101
(2) all other categories can be related to it; and
(3) it appears frequently and consistently in the data.
The findings and discussion section presents the model for elementary teachers’ learning
experiences as they navigated a professional development literacy website. The findings and
discussion section also describes the nine major themes and includes direct quotes to support
each theme. Preceding the findings and discussion of study 2, the remaining methods section
highlights the strategies and techniques used for establishing trustworthiness of the qualitative
findings.
Establishing Trustworthiness
Lincoln and Guba (1985) provide a framework for establishing trustworthiness in their
book, Naturalistic Inquiry. This framework guides researchers in establishing rigor of
qualitative findings (Agostinho, 2005; Corbin & Strauss, 2008; Glesne, 1999; Golafshani, 2003;
Leech & Onwuegbuzie, 2007; Shenton, 2004) by outlining four strategies of trustworthiness
(credibility, transferability, dependability, and confirmability), and providing techniques to
enhance the trustworthiness of qualitative findings. Following Lincoln and Guba’s (1985)
framework, study 2 achieved trustworthiness through the following six techniques: (1) member
checking, (2) triangulation, (3) frequency tallies, (4) thick descriptions, (5) compiling an audit
trail, and (6) screen shots of the analysis process. The remainder of this section describes each
technique under the headings: credibility, transferability, dependability, and confirmability.
Figure 4.2 outlines the techniques used to achieve trustworthiness of the findings.
Establishing Trustworthiness
1. Credibility
a. Member checking
b. Triangulation
c. Frequency of categories across participants
102
2. Transferability
a. Thick descriptions
3. Dependability
a. Audit Trail
4. Confirmability
a. Triangulation
b. Audit Trail
Figure 4.2. Topics pertaining to establishing trustworthiness
1. Credibility. The first criterion for establishing trustworthiness is credibility—the
believability of the findings and interpretations; it is the truth value and validity of the findings
(Lincoln & Guba, 1985). To enhance the credibility of the findings and interpretations of study
2, three techniques were used: member checking, triangulation, and tallying the frequency of
categories across participants.
a. Member checking. Member checking has been suggested as “the most crucial
technique for establishing credibility” (Lincoln & Guba, 1985, p. 314). As previously
mentioned member checking occurred during the axial phase of analysis. The process involved
the participants’ feedback on seven preliminary themes and is described above. Member
checking contributed to the credibility of the findings by verifying the validity of the themes
through participant feedback.
b. Triangulation. Triangulation involves the convergence among multiple sources of
information (Golafshani, 2003). According to Lincoln and Guba (1985) four different modes of
triangulation exist—the use of multiple and different sources, methods, investigators, and
theories. The current study collected and analyzed three different sources of data—think alouds,
interviews, and questionnaires. Analyzing the interview and questionnaire data, in addition to
the think aloud transcripts, helped establish the consistency and frequency of categories beyond
the think aloud transcripts (see Appendix C-5). Furthermore, the interview allowed participants
103
to expand upon their think aloud and answer direct questions about their website navigation.
Because prompts, questions, and redirections were limited during the think aloud, the post-think
aloud interview provided opportunities for participants to expand upon their ideas and follow up
on their decision making strategies. Triangulation contributed to the credibility of the findings
by adding depth to the findings. Selected quotes from the three data sources are included in the
results and discussion section.
c. Tallying the frequency of categories across participants. A frequency chart was
developed to determine the frequency of the categories across participants. As previously
stated, categories across participants were tallied for each data source. This process contributed
to the credibility of the findings by confirming the consistency of the final categories across
participants.
2. Transferability. The second criterion for establishing trustworthiness is
transferability—the potential applicability of the results to other contexts. Transferability
depends on the degree of similarity between the original research context and contexts in which
the results may be transferable (Lincoln & Guba, 1985). Lincoln and Guba (1985) caution
qualitative researchers that “the naturalist can only set out working hypotheses together with a
description of the time and context in which they were found to hold” (p. 316). In qualitative
research, transferability is not about generalizing the findings; rather transferability is about
“providing the database that makes transferability judgements possible” (p. 316). To enhance
transferability in the current study, thick descriptions were used.
a. Thick descriptions. In this study, thick descriptions include details of the context and
study participants, and direct quotes and interpretations presented in the findings and discussion
section. That is, participants’ actions and experiences are “thickly described…so that thick
interpretations of the actions can be made, presented in written form, and be made available to a
104
wide audience of readers” (Ponterotto, 2006). Thick descriptions enhance the potential
transferability of the findings. That is, descriptions of the website used in the current study will
allow future researchers to undertake similar studies using any professional development
website across a range of domains. In addition, extensive participant quotes can provide
researchers with a basis of the think aloud transcripts and units of analysis used in this study.
3. Dependability. The third criterion for establishing trustworthiness is dependability—
the consistency of the findings. Dependability is typically demonstrated by replication (Lincoln
& Guba, 1985). That is, if two or more studies yield similar findings then the reliability of the
findings has been established. Future studies attempting to replicate the findings of the current
study must consider the process of data analysis; therefore, the current study enhanced the
dependability of the findings through the compilation of an audit trail.
a. Audit trail. The audit trail in Appendix C-6 includes samples of raw data and a sample
screen shot from NVivo that shows the nine major themes. In addition, the audit trail includes
figures of the theory construction. An audit trail contributed to the dependability of the findings
by documenting the development of the analysis, and providing, as much as possible, an account
of the research decisions and activities throughout the study (Carcary, 2009).
4. Confirmability. The last criterion cited by Lincoln and Guba (1985) is
confirmability—ensuring, as much as possible, that the findings emerged from the data and not
the predispositions of the researcher (Shenton, 2004). Confirmability was achieved in this study
through triangulation and the audit trail. As previously described, these techniques provide the
reader with information about the consistency of the findings (triangulation) and the research
activities throughout the course of the study (audit trail). In addition, the primary investigator
was also aware of the potential overlap in categories between study 1 and study 2. While it may
be impossible to reduce researcher bias completely, the primary investigator approached the
105
analysis of study 2 with an open mind and attempted to code units of analysis by staying as
close to the data as possible.
In sum, several techniques were employed to enhance the trustworthiness of qualitative
findings: (1) member checking, (2) triangulation, (3) frequency tallies, (4) thick descriptions, (5)
compiling an audit trail, and (6) screen shots of the analysis process. While these techniques
have been cited by many qualitative researchers as strategies to enhance the trustworthiness of
the qualitative findings, a cautious interpretation of the findings is recommended since the
current study is context-specific and exploratory in nature.
Findings and Discussion
The previous section described the methods of study 2, including participant
demographics, the research design process, data analysis, and techniques for establishing
trustworthiness. This section presents the findings and discussion of study 2. As previously
described, the primary purpose of study 2 was to gain an in-depth understanding of elementary
teachers’ learning experiences as they navigated a professional development website. The
findings presented in this section represent the participants’ learning experiences as they used
The Balanced Literacy Diet website. These findings are organized according to nine major
themes that resulted from the analysis. Table 4.2 presents a description of each theme and an
example of each theme from the think aloud transcripts.
Table 4.2
Study 2 Themes
Theme Description Example from the Transcripts
1. Perception of
Professional
Learning
Ideas and attitudes related to professional
learning and literacy instruction,
including credibility and trustworthiness,
and research-based practices.
“If the information is accurate and
well-presented, teachers will go
back to the website. There are so
many websites, there’s so much
information...I need to know the
information here is valid.”
2. Focusing on
Student Needs and
A connection in reference to a teaching
context, students, and current and future
“I’m gathering ideas and
information that are relevant to
106
Instructional Goals
instructional practices. my students’ needs and interests.”
3. Individual
Differences
Factors relating to motivation, online user
behaviour, prior experience, and
professional knowledge.
“I find it really helpful because I
can see what the classrooms are,
and sometimes there are lessons
and links that I can look into
which helps me as a learner and as
a teacher.”
4. Navigating a
Professional
Development
Website
Descriptions and explanations of website
actions and behaviours based on
participants’ exploration and
understanding of the website.
“First I was trying to get a general
sense of the website and what was
on it.”
5. Evaluating
Information
Evaluative statements (assessments,
opinions) in relation to participants’
navigation.
“It was nice also that you could
sort by grade and then re-filter by
the food groups.”
6. Encoding
Information for
Future Retrieval
Strategies for remembering and recalling
information including note taking,
bookmarking, and emailing.
“I definitely at this point wanted
to record the title of this book, so
I’m going to open up a document
so I can start typing out my
ideas.”
7. Reflecting on
Learning
Reflective statements about learning from
the website and seeing one’s self as a
learner, including feelings towards the
website, future use of the website, and
navigational choices.
“As a new teacher it’s really
important to go over things and
see examples of what people
expect.”
8. Continued
Professional
Learning
Statements relating to the advantages of
online professional learning, increased
self-efficacy, feeling inspired, and
knowledge construction.
“Here I’m just having a look at
this classroom…and it can remind
you of things, it can show you
things that you haven’t thought of
and give you inspiration and new
ideas.”
9. Intentions for
Practice
Statements about planning, adapting
information to meet student needs, and
professional goals.
“I can tell with this activity that
it’s multi-layered, it can be turned
into a whole unit and it can allow
me to develop many different
skills, literacy skills and other
skills with the students.”
107
The nine themes listed in Table 4.2 were organized into a theoretical model based on the
current study’s research questions as well as four components of the grounded theory
framework (Strauss & Corbin, 1998). Figure 4.3 presents the theoretical model for elementary
teachers’ learning experiences as they navigated a professional development website. The
remaining chapter presents and discusses each theme and pertinent topics under the following
headings: (1) conditions affecting elementary teachers’ website navigation, (2) the central
phenomenon resulting from the conditions, (3) strategies for navigating a professional
development website, and (4) potential outcomes resulting from elementary teachers’ website
navigation. Direct quotes are included to support each theme.
Elementary Teachers’ Learning Experiences as they Navigate a
Professional Development Website
Conditions Affecting Central Phenomenon Strategies for Navigating
Navigation a Professional Development
Website
Theme 1:
Perception of
Professional Learning
Theme 4:
Navigating a
Professional
Development Website
Theme 5:
Evaluating
Information
Theme 2:
Focusing on Student
Needs and Instructional
Goals
Theme 6:
Encoding
Information for
Future Retrieval
Theme 3:
Individual
Differences
Potential Outcomes of Navigating a Professional Development Website
Theme 7:
Reflecting on Learning
Theme 8:
Continued Professional
Learning
Theme 9:
Intentions for Practice
Figure 4.3. Theoretical model for elementary teachers’ learning experiences as they navigated a
professional development literacy website.
108
Conditions Affecting Elementary Teachers’ Website Navigation
The first three themes of the theoretical model are the conditions affecting elementary
teachers’ website navigation. These themes emerged as a result of the analysis and include:
Theme 1: Perception of Professional Learning; Theme 2: Focusing on Student Needs and
Instructional Goals; and Theme 3: Individual Differences. Each theme and pertinent topics are
discussed separately below.
Theme 1: Perception of Professional Learning
In the current study, Theme 1: Perception of Professional Learning refers to the
participants’ ideas, attitudes, and beliefs related to professional learning resources and
instructional practices. Participants’ thought units were classified through the data analysis into
two main groups: (a) quality of professional resources, and (b) beliefs about literacy instruction.
Figure 4.4 presents the topics pertaining to the first theme.
Theme 1: Perception of Professional Learning
1. Quality of Professional Resources
a. Website authenticity
b. Background information
c. Source of recommendation
2. Beliefs about Literacy Instruction
Figure 4.4. Topics pertaining to theme 1: Perception of professional learning
1. Quality of Professional Resources
In the current study, participants identified three source features that they believe can
affect the credibility, trustworthiness, and overall quality of a given professional resource: (a)
website authenticity—whether the content is believable and reflects realistic contexts and real-
life strategies, (b) background information—the expertise of the developers and supporting
organization, and (c) source of recommendation—such as, a trustworthy colleague or
109
administrator. All of the participants felt confident in their use of professional resources with
the presence of these source features.
a. Website authenticity. Participants were willing to trust the information on the given
website because it included authentic contexts and realistic examples—the information appeared
believable. For example, one participant commented on the authenticity of the virtual
classrooms: “I think it’s a great website, I think that there’s so much value on it because it’s
authentic in the fact that you can see into people’s classrooms.” In addition, a participant who
viewed an activity related to vocabulary growth thought “it was interesting in that they found
one specific word in an article it looks like, and they used that as their challenging word. So
that’s the teacher showing a real life example.” Similarly, another participant described in an
interview how “the real life photographs and images of the different parts were the most eye-
catching…it was a real classroom, it wasn’t a theoretical place.”
The above examples suggest that elementary teachers value professional information that
accurately reflects their own practice. By viewing real-life examples of classroom environments
and authentic teaching practices, elementary teachers can make connections between new
information and their own practice as well as visualize possible future activities. Moreover,
real-life examples can contribute to a heightened teacher self-efficacy—teachers gain
confidence in their own teaching abilities when they view realistic models to which they can
relate.
b. Background information. Along with authentic examples, participants noted how
the background information—for example, the supporting organization—affects a website’s
credibility and overall quality. For example, one participant described her thoughts about the
credibility of the given website and included her interests in understanding the background
information:
110
I was still reading about the Institute’s mission and what its goals were. And I guess I’m
doing this to make sure that the website’s credible and that maybe if nothing jumps out
that contradicts my own pedagogical practice or my own ideas around teaching
literacy…And then I was interested in who was sort of behind the Institution so I clicked
on the ‘about us’ button and read a little bit about the history about why the Institution
was set up. And then I scrolled down to read about the members of the board.
As a result of gaining information about the website’s background, this participant determined
whether the website corresponded to her own practice and beliefs about teaching literacy.
Another participant acknowledged the website’s university association during an
interview: “I like that [the website is] from the [university] site, because I think that does give it
credibility.” Again, information about the website’s developers and supporting organizations
contributed to participants’ willingness to trust the information presented on the given website.
The background information was easily accessible and offered the participants a sense of ease in
their quest for professional knowledge and materials.
c. Source of recommendation. The source of recommendation (a colleague or school
administrator) also appeared to affect the website’s credibility. For example, as one participant
viewed the screen recording of his navigation he wondered about the credibility of the
supporting organization, one with which he was unfamiliar:
I’m also wondering what the Melissa Institute is, because I didn’t go here, you know I
didn’t navigate here from somewhere else in a way that makes sense to me. Or I didn’t
hear about this particular website from someone that said, ‘hey you really got to check
this out’; you know, from a colleague or from a workshop of some kind.
Another participant had a similar concern: “Because no one recommended [the website] to me,
there’s the element of trust missing initially.” Similarly, in a questionnaire response a
111
participant presented the benefits of using online professional resources and reasons why she
favoured school-board websites: “It brings together links to resources in one easily accessible
place…the links have already been checked for authenticity and are ‘Board Approved’ and this
takes the leg-work out of the process for me.” These examples suggest that teachers value their
colleagues’ recommendations and opinions about professional learning. In turn, source
recommendations appear to influence teachers’ use of a professional resource.
Taken together, the website’s authenticity, background information, and source of
recommendation appear to affect the credibility, trustworthiness, and overall quality of the given
website. Over half of the participants in the current study sought information related to the
website’s credentials; some participants noted the journal citations on various web pages; and
participants stated that in general, credibility and trustworthiness impacts their future use of any
type of professional development resource.
It is well-known that online resources vary in terms of the quality of information they
provide. Research that has investigated the trustworthiness of online resources is based on
university student and high school student responses (Iding, Crosby, Auernheimer, & Klemm,
2009; Wiley et al., 2009). Findings of such studies indicate that many Internet users interpret
“superficial or inadequate source features (e.g., popularity)” when judging a website’s
trustworthiness (Andreassen & Braten, 2013, p. 823). The findings pertaining to theme 1of the
current study reveal the use of effective source features by elementary teachers when judging a
website’s trustworthiness. Elementary teachers want to ensure that their professional knowledge
and instructional materials stem from credible sources, and examining these source features
during professional learning enhances their confidence in the resource.
112
2. Beliefs about Literacy Instruction
In the current study, all of the participants held positive beliefs about research-based
practices in literacy and often referred to the components of effective literacy instruction.
Specifically, participants perceived The Balanced Literacy Diet website as believable and
expert-driven because it presented “good evidence of modern research and instructional
strategies.” The website presented “great ideas because they connect to the research and how
that’s all supported.”
One participant stated in the questionnaire that she “thinks it’s important to keep up with
best practices in education,” and that she enjoys “interacting with experts and other teachers” to
gain knowledge about meaningful teaching methods. Additionally, participants described the
importance of staying “current to learn about innovative teaching practices.” Participants also
sought innovative strategies that align with their views of literacy instruction. For example, one
participant stated that he “was happy that there were good examples of criteria lists…its
reinforcing effective discussion within the classroom and [the website] reinforces the concepts
of what the ministry and the literacy/numeracy secretariat is trying to promote.”
Participants referred to several other research-based literacy practices during their think
aloud. For example, one participant referred to the importance of including a variety of print
materials in the learning environment: “Remembering that the balanced literacy program
includes fiction, non-fiction, all types of literature to make sure that, you know, kids are getting
a good variety.” In addition, a participant described the practice of teacher-student
collaboration:
So it’s really important to me that the students are clear on the learning goals of each
subject area of the class. We brainstorm together what the learning goals will be in each
subject and the students actually come and write the learning goals with me. They
113
become a reference point for them if they are ever wondering what we’re working on
they can easily come to the board.
Participants also referred to:
Differentiated instruction—“I liked the idea that it was reinforcing differentiated
instruction and having books at different levels,”
Motivation for literacy through games—“I like this idea of, you know, connecting new
vocabulary with a game and some sort of fun activity that we can do with it,”
Integrated instruction—“And I liked again, the link to a different subject area, so linking
that to social studies and science, I thought it was really great that she combined her
math with language,” and
Student accommodations—“I also like the idea of colour coding. That’s something that I
should do more. I think it can support some of my students with learning
disabilities…some students I think would find it helpful, when we’re writing paragraphs
or any sort of writing that can be kind of compartmentalized to learn the form.”
The above examples demonstrate participants’ positive beliefs about research-based
practices. Many of these examples are consistent with the evidence-based components of high
quality literacy instruction (Gambrell & Anders Mazzoni, 1999; National Reading Panel, 2000;
The National Early Literacy Panel, 2008). That is, there is a general agreement of what
constitutes best practices in literacy instruction (see Table 4.3). The findings pertaining to
theme 1 suggest that teachers are cognizant of these practices when they approach professional
development websites.
114
Table 4.3
General Agreement of What Constitutes Best Practices in Literacy Instruction
Best Practices in Literacy Instruction
Teach reading for authentic meaning-making literacy experiences
Use high-quality literature,
Integrate a comprehensive word study/phonics program into reading/writing instruction,
Use multiple texts that link and expand concepts,
Balance teacher- and student-led discussions,
Give students direct instruction in decoding and comprehension strategies that promote
independent reading,
Use a variety of assessment techniques to inform instruction,
Explicit instruction in reading skills, including letter-sounds, phonics, spelling, vocabulary,
word recognition, and comprehension skills
Note: From Gambrell & Anders Mazzoni, 1999; National Reading Panel, 2000; The National
Early Literacy Panel, 2008.
Taken together, the findings pertaining to theme 1 suggest that elementary teachers
approach professional learning with particular beliefs, values, and ideas. In turn, their
perception of professional learning may affect the short- and long-term use of an online
professional resource.
Theme 2: Focusing on Student Needs and Instructional Goals
As teachers approach online resources they carefully reflect on the relevance and
applicability of the given resource. All of the participants in the current study sought
information specific to their practice; they were mindful of the current and future needs of their
students and their instructional goals. Theme 2: Focusing on Student Needs and Instructional
Goals is the second theme that emerged from the analysis and consists of the following topics:
(1) student needs, (2) teaching context, and (3) current and future instructional goals. Figure 4.5
presents the topics pertaining to the second theme.
115
Theme 2: Focusing on Student Needs and Instructional Goals
1. Student Needs
2. Classroom Context
3. Current and Future Instructional Goals
Figure 4.5. Topics pertaining to theme 2: Focusing on student needs and instructional goals.
1. Student Needs
As participants navigated The Balanced Literacy Diet website their web-based actions
and behaviours were affected by the needs of their current students. Participants were interested
in “gathering ideas and information that [were] relevant to [their] students’ needs and interests.”
For example, one participant stated: “when I was reading the different recipe elements I was
specifically looking for key words that target some of the things I do in my special education
classroom.” As this quote demonstrates, participants were consistently referring to their
students’ needs and interests during the 20-minute website navigation. Participants had
individual interests as they sought instructional materials. This finding suggests that teachers
value professional resources that support their students.
2. Classroom Context
Participants also based their navigational decisions on their classroom context. For
instance, one participant stated:
I teach grade one/two, so clicking on the first to second grade virtual tour and just having
a look at the classroom it shows and then looking down below to see some of the
particular recipes I guess that are in that tour…And looking at just the little snapshots
and the description underneath for anything I might be interested in.
This participant chose topics directly relating to her individual classroom context. In this sense,
she was self-directing her learning. This finding suggests that teachers who are given the
116
opportunity to choose pertinent topics are likely to gain relevant information with immediate
applications.
3. Current and Future Instructional Goals
As participants navigated the website, they also reflected on their current and future units
of study. For example, as one participant viewed a photo gallery of an integrated math and
literacy activity, she stated: “we’re looking at symmetry right now, so that seemed really
interesting.” Similarly, as another participant viewed a recipe about literature circles she
referred to a future instructional goal: “I’m wanting to start literature circles with my class, like
maybe this week or next week, so here I’m having a look at literature circles and just reading
through this one in some detail.” In addition, while navigating the given website, a participant
focused on a programming goal: “I’m hunting for anything related to grade two and lately I’ve
been wanting more word work activities…so I’m looking for any word work or sight words.”
In sum, participants in the current study sought information that directly applied to their
students’ needs, classroom context, and instructional goals. This finding is consistent with two
characteristics of the adult learner: self-directed learning, and immediate application of
knowledge. Given that the participants were directing their navigation themselves, they were
free to choose topics that best fit their needs and current context. There was also a sense of
immediate application—participants had a vision for their practice and recognized that the
information could be immediately applied to their current and future units of study.
Theme 3: Individual Differences
Individual Differences is the third theme of the current study’s theoretical model.
Individual differences were classified through the data analysis into three categories: (1)
motivating factors to use online resources, (2) online user behaviour, and (3) prior experience
and professional knowledge. Figure 4.6 presents the topics pertaining to theme 3.
117
Theme 3: Individual Differences
1. Motivating Factors to Use Online Resources
a. Ease of use
b. Personal interest to expand professional knowledge
2. Online User Behaviour
a. Forms of communication
b. Multitasking behaviours
c. Type of learner
3. Prior Experience and Professional Knowledge
a. Professional challenges
b. Area of expertise
Figure 4.6. Topics pertaining to theme 3: Individual differences.
1. Motivating Factors to Use Online Resources
Teachers are generally motivated to use online resources for their professional learning
(Charalambousa & Ioannou, 2011). In the current study, 11 participants referred to two factors
that contribute to their motivation to use online professional resources: (i) the ease of use of
online resources, and (ii) their personal interest to expand professional knowledge.
a. Ease of use. Participants agreed that the ease and accessibility of the Internet is a
motivating factor to use professional development websites. For example, one participant
preferred the convenience of online resources as opposed to professional texts:
I find [professional development websites] really convenient…instead of going to the
library and taking out books, things are already narrowed down for you and the
important things are already there, in a sense that you don’t have to search and waste so
much time…it’s a lot easier.
Similarly, another participant stated that “the Internet is very portable these days…you can use
it on your phone, outside the house…it’s very convenient to use online resources.” In addition,
a participant described the convenience of the Internet over a professional library of texts and
118
books: “I find [the Internet] is much nicer than having to collect books and having a huge library
of books…sometimes you might only want a small segment of it or something like that…so I
appreciate having access to online resources.”
These examples suggest that access to professional development websites offers
information that teachers may not as easily access through more traditional face-to-face
professional development activities. This is consistent with other research that has examined
teachers’ motivations towards online learning (Jones & Dexter, 2014; Recker et al., 2007).
Specifically, teacher reports have indicated that informal online environments are flexible in
terms of time and location, and can provide on-going and “just-in-time” support (Jones &
Dexter, 2014). These just-in-time supports can include accessible and easy-to-use professional
development websites—teachers can expand their professional knowledge and repertoire of
instructional strategies at their own time and convenience.
Additionally, online professional resources make it easier for teachers to easily and
immediately transfer material to their practice. This is consistent with one of the characteristics
of the adult learner: adults value material that they can easily and immediately transfer to their
practice. Professional development websites offer elementary teachers with timely professional
knowledge and instructional material; a teacher might wish to immediately follow up on her
students’ learning, and can quickly search for relevant supports and strategies.
b. Personal interest to expand professional knowledge. Participants referred to their
motivation to expand their repertoire of professional knowledge by seeking out research-based
practices. For example, one participant, who identified herself as a new teacher, stated in a
questionnaire response “that it is important for [her] to continue learning new strategies
(especially in the area of literacy).” She reported that “conferences and workshops are inspiring
to go to…they provide new and creative ways to teach.” Similarly, another participant reported
119
that “it’s important to keep up with best practices in education.” This participant stated that she
“enjoys interacting with experts and other teachers,” and that she is “always looking for new
ideas to bring back to [her] classroom.”
This finding is consistent with the characteristics of adult learners. As previously
discussed, teachers’ willingness to engage in professional development increases when the
material is relevant and personally meaningful to them (Chametzky, 2014). Professional
development websites lend themselves to greater learner control and autonomy, and ultimately
intrinsic motivation to learn. As a result of self-directing their own learning, teachers may be
more willing to implement best practices in the classroom.
2. Online User Behaviour
The second individual difference found as a result of the analysis is differences in
participants’ online user behaviour. Three main differences in online user behaviour emerged
from the analysis: (i) forms of communication, (ii) multitasking behaviour, and (iii) types of
online learners.
a. Forms of communication. Participants in the current study most often referred to two
forms of communication: visual media—visual representations of information—and
multimedia—a form of communication that includes more than one type of media element
(Mayer, 2002). Both forms of communication appeared to play a role in participants’
navigational experiences. Specifically, three participants described how visual media, including
photo galleries and virtual tours, contributed to their online learning experience. For example,
one participant stated:
I find [visual tools] really helpful because I can see what the classrooms are, and
sometimes there are lessons and links that I can look into which helps me as a learner
and as a teacher, so I can be like, ‘oh this is a good idea, maybe I can use in my class’.
120
Similarly, a participant indicated: “I’m always looking at visuals and how things are organized. I
get a lot visually from websites. I like the layout of the movie and the print. All of this is a nice
design.” In addition, a participant verbalized her thoughts about visual media as an important
form of communication during planning: “I find it really important to see actual examples and
that’s what I tend to do when I’m using online resources to help me plan my lessons.”
One participant who accessed a video of an exemplary teacher describing a lesson thinks
aloud about the benefits of this audio-visual modality:
Wow, this is really interesting—that there are videos too. It’s really helpful to me
because that’s the best way I like to see things—get a really quick look at what’s going
on, and what they have to say.
Furthermore, three participants described how the videos fostered a personal approach to
professional learning and that this personal approach contributed to the retention of information.
For example, one participant referred to the personal approach taken with the videos:
It’s neat because it feels like she’s actually talking to me which makes it personal in a
way…instead of just looking at pictures on, let’s say Pintrest of anchor charts, she
explains to me what is expected and the steps that you can do…if you just looked at a
photo you wouldn’t understand it, but there’s an explanation to it which is very
beneficial.
Taken together, participants appeared to prefer two forms of communication during their
20-minute website navigation: visual media and audio-visual modalities. Additionally, many
participants linked their use of multimedia forms of communication to their ability to understand
and retain information. This finding is consistent with research that has examined the use of
multimedia design and memory. Specifically, recent research findings indicate multimedia
designs that incorporate audio-visual modalities promote recall and retention performances
121
(Mutlu Bayraktar & Altun, 2014). This suggests that when professional development websites
contain elements of multimedia, teachers may be more likely to create memorable learning
experiences.
b. Multitasking behaviour. Half of the participants described multitasking behaviour.
In particularly, one participant “figured that [she] could probably start typing and [she] was able
to listen to the video at the same time and start [her] document.” Similarly, another participant
suggested that when using online resources “you’re always multi-tasking.” She went on to
explain:
Even though I’m listening, I’m still looking around…I’m thinking about all the other
choices or taking in more information...I like the way [the video] can just be played at
the side, and then you can be reading about the next thing that you’re going to go to.
According to the demographic data, the participants who multitasked during their 20-
minute website navigation ranged in terms of age and years of experience. In addition,
participants who multitasked during their navigation indicated that they felt very comfortable
using the Internet and technology for professional purposes. These findings suggest that
comfort with technology may be related to website use, whereas age may not be related. This is
consistent with the third characteristic of the adult learner—the use of experience as a source of
information. As teachers’ experience with online learning environments develops their comfort
with technology also develops; thus, the way they engage with a given website changes over
time.
c. Type of online learner. Eleven participants also identified themselves as specific
types of learners. Some participants described themselves as “skimmers” where they scan the
web pages first before focusing on their objectives. For example, one participant “really liked
the transcript feature because [she] didn’t feel like [she] needed to watch the video, so it was
122
good to skim.” Another participant identified herself as “a digital native.” As a digital native,
web users “whip through things pretty quickly.”
In sum, the findings pertaining to the topic, online user behaviours can be considered in
the design of professional development websites. That is, professional development resources
for teachers should be geared towards a broad audience, to both experts and novices. In
addition, the design should include a variety of visual and multimedia web-based tools.
Moreover, professional development that incorporates informal online resources should allow
for multiple experiences with the technology over time. Taking these aspects of design into
consideration can contribute to elementary teachers’ willingness to engage as active learners and
to therefore transfer their knowledge to practice. Additionally, these considerations may
influence teachers’ continued use of high quality professional development websites.
3. Prior Experience and Professional Knowledge
The third individual difference that emerged as a result of the analysis is prior experience
and professional knowledge. Two main differences were found with respect to participants’
prior experience and professional knowledge: (i) professional challenges, and (ii) area of
expertise.
a. Professional challenges. Five participants described professional challenges they
encounter when they develop their programs. For example, one participant described the
challenge of developing and implementing a junior level reading program:
“I find it tough teaching, like developing and teaching a reading program to grades five
six…it’s easy to think that in grades five and six you’re done teaching reading, and of
course we’re not…I’m still struggling with figuring out exactly how my reading program
should look and I find that reading gets neglected in favour of writing.”
123
This example demonstrates how this participant reflects on her professional challenges as she
uses a professional development website. As a result, she is likely to seek out information that
will aid her in program planning.
b. Area of expertise. Participants’ navigational choices also appeared to be influenced
by their area of expertise with respect to curriculum content. As one participant viewed an
integrated arts lesson, she identified her qualifications: “I have a visual arts background… I’m
also a special education teacher, so this was a good amalgamation of my two interests.”
Another participant referred to the obtainment of a recent additional qualification and how this
influenced her choice to view Kindergarten material: “I thought I might check out the earlier
grades. I took a Kindergarten AQ last year and I’m really interested in full day kindergarten…so
I thought I’d take a look and see how that was.”
Taken together, these findings demonstrate that some participants used experience as a
source of information. For instance, in the former example, the participant reflected on her
experience and recognized her professional challenges. She then attempted to move beyond her
professional challenges to refine her reading program. In the latter examples, the participants
refer to their expertise and make connections between their knowledge and related instructional
methods.
In sum, individual differences appeared to affect how participants navigated The
Balanced Literacy Diet website. Specifically, the findings pertaining to theme 3 suggest that
elementary teachers demonstrate several differences as they approach and utilize online
professional development websites: elementary teachers show differences in their motivation to
use online resources; elementary teachers vary in their user behaviours and prefer specific forms
of communication for their learning; elementary teachers view themselves as a particular type of
124
online learner; and prior experience and professional knowledge appear to affect how
elementary teachers navigate professional development websites.
The significance of the findings pertaining to theme 3 is twofold: (1) the findings
emphasize elementary teachers as individual learners, and (2) the findings identify specific
differences in elementary teachers’ use of a professional development website. These findings
can be taken into consideration in the design of professional development websites—the design
should be geared towards a broad audience and various tools should be accessible.
The Central Phenomenon Resulting from the Conditions
The previous section described the first three themes of the theoretical model:
Elementary teachers’ learning experiences as they navigate a professional development website.
Specifically, perceptions, foci, and individual differences emerged as the conditions affecting
participants’ website navigation. This section presents Theme 4: Navigating a Professional
Development Website. As previously mentioned, theme 4 was deemed the core or central
phenomenon of the theoretical model because it represents the main theme of the research, all
the other categories can be related to it, and it appeared frequently and consistently in the data.
Figure 4.7 presents the topics pertaining to Theme 4: Navigating a Professional Development
Website.
Theme 4: Navigating a Professional Development Website
1. Gaining a General Sense of the Website
2. Exploring In-Depth
3. Understanding the Website
Figure 4.7. Topics pertaining to theme 4: Navigating a professional development website.
Theme 4: Navigating a professional development website
While there were differences in how the participants approached and used the website
(i.e., motivational factors, foci, area of expertise), there was a general pattern of navigation for
125
all participants as they used The Balanced Literacy Diet website. This navigational pattern
followed three stages: (1) gaining a general sense of the website, (2) exploring in-depth, and (3)
understanding the website.
1. Gaining a General Sense of the Website
As participants began their website navigation they were interested in gaining a general
sense of the website’s framework, the web pages, and the various web-based tools. The
participants considered these elements to “get a general sense of the website and what was on
it,” and to “explore the recipes, the lessons, and understand the thinking behind the website.”
During their initial navigation, participants were interested in “how the recipes were created or
the lessons were laid out.” For example, one participant described his web-based behaviour
when approaching any new website.
I think initially when you come to a new website you want to be able to play around,
move around through your anxiety, put it aside and then soon after get a sense of what’s
there and how to navigate, and how many layers there are. Then I think you can get into
it more and relax more…I was able to do that.
Participants were generally interested in gaining “more of a feel for what these recipes or
lessons were before using the recipe finder and starting to actually lose [themselves] in reading
the recipes more specifically.” As one participant viewed a classroom virtual tour, she explained
that she was “just trying to take everything in, to see how all the different subjects are integrated
in the classroom, how things are being displayed, and just the use of wall space and shelf
space.”
2. Exploring In-Depth
As participants maneuvered through the website, they began to “explore the website in a
little bit more depth” and connect and plan according to their current practice, classroom
126
context, and student needs. For example, during one participant’s think aloud she stated: “At
this point I was trying to read through the transcript. I was trying to figure out, make sure I had
all the details for that because it was an activity I thought I could possibly use.” Similarly,
another participant decided “to go in and read in a little bit more detail and read about the oral
language food group, how it’s described by the website.” In addition, a participant reflected, “In
the beginning I was more focussed on how to use it or what was available and now I’m actually
almost, like thinking about how to put it into practice.”
3. Understanding the Website
As participants continued to explore the website in depth they began to understand the
features and underlying architecture of the website. For example, as a participant viewed the
screen recording of her navigation she thought back to navigational choices and “realized that
you can actually scroll by teacher…I said, ‘oh you can actually see things by teacher,’ so you
actually might find favourite teachers that you like to follow.”
Participants also developed an understanding of the organization of the website. For
example, one participant reflected on how he was “discovering the way the website’s organized,
at the bottom of each page describing each food group. I can also use that as a filter to find
activities.” Similarly, another participant grasped the workings of particular web-based tools: “I
figured out that if you rested on the red dot you could find out what the [learning] centre was. So
I started reading more of them…there’s so much in there.” In addition, a participant described
her process of understanding how the virtual tour worked as a professional tool:
And I wasn’t too sure, I mean I’m not too familiar with virtual websites so I wasn’t too
sure what was going to happen. I thought I was just going to visit the classroom but then
I realized that as I clicked on things I got information, like what is displayed right now
127
about the chart. And then I think I went back and clicked on one of the red dots and
realized that this little video popped up which was kind of neat.
In sum, participants’ 20-minute navigations followed a pattern of three stages. Initially,
participants were interested in gaining a general sense of the website. This general overview
was followed by an in-depth exploration during which time participants were likely to relate
website content to their teaching practice. As participants explored in-depth they developed a
greater understanding of the website features and various web-based tools. This general pattern
of web seeking behaviour has implications for educational website developers. Understanding
teachers’ website navigational patterns can contribute to the development of online resources
that best support users’ online behaviours. Online learning platforms can be designed in a way
that easily permits first time users to navigate web pages as well as link back to the homepage or
central hub of the resource. By gaining a general sense of the online resource, the user can
decide to continue to explore in-depth.
Strategies for Navigating a Professional Development Website
The previous section described Theme 4: Navigating a Professional Development
Website and highlighted a pattern of three stages of a participants’ website navigation. This
section describes two strategies that participants employed while navigating a professional
development website. These strategies emerged as a result of the analysis and include: Theme
5: Evaluating Information; and Theme 6: Encoding Information for Future Retrieval. As
participants navigated The Balanced Literacy Diet website they employed these strategies to aid
them in their learning and their use of the given website. Each theme and its relevant topics are
discussed separately below.
128
Theme 5: Evaluating Information.
Evaluation involves the collection, organization, and analysis of information. In the
current study, participants evaluated a range of components during their 20-minute web
navigation. These components coincide with six criteria for evaluating informal online learning
environments as identified by Song and Lee (2014): content richness, functionality of the
technology, extent of technology integration, novelty of the technology, uniqueness of the
learning environment, and potential for learning (see Table 4.4 for a summary of Song’s
criteria). Figure 4.8 presents the topics pertaining to Theme 5: Evaluating Information.
Theme 5: Evaluating Information
1. Content Richness
2. Functionality of the Technology
3. Extent of Technology Integration
4. Novelty of the Technology and Uniqueness of the Learning Environment
5. Potential for Learning
Figure 4.8. Topics pertaining to theme 5: Evaluating information.
Table 4.4
Song and Lee’s Criteria for Evaluating Informal Online Environments
Criteria Description
1. Content richness Adequacy of content for learning purposes, credibility and currency
of the information
2. Functionality of the
technology
Ease of access and navigation
3. Extent of
technology
integration
Wide range of technologies, interactive and collaborative tool
4. Novelty of the
technology
Utilization of emerging technology for learner excitement and
motivation
5. Uniqueness of the
learning
environment
Opportunities to explore real-world issues through authentic
learning experiences
129
6. Potential for
learning
Focus on achievement of learning goals, tools for tracking learning
Note: From Song & Lee, 2014
1. Content Richness
Participants evaluated the content richness of the given website—“the adequacy of
content for learning purposes, credibility and currency of the information” (Song & Lee, 2014).
For instance, one participant stated:
It felt like it was a very comprehensive website, so it had a lot of information and also
that it had a lot of practical teaching strategies, so really useful things like seeing a real
classroom and hearing that teacher speak to her classroom, I found that really useful.
The quality of the content contributed to the participants’ learning experience by
ensuring the trustworthiness of the information presented on the website. Zaharias (2009)
emphasizes that the basis of an online learning environment “should be the quality of content”
(47). Content should not only be relevant, but it should also come from reliable and cited
sources (Zaharias, 2009).
2. Functionality of the Technology
Participants also evaluated the functionality of the technology—“the ease of access and
navigation” (Song & Lee, 2014). For instance, a participant described his “general
feelings…that [the website] was a relatively clean and well-organized…there were lots of
headings and boxes and colours and fonts but it wasn’t overwhelming…there was a consistent
colour scheme which I think is very helpful.” The website was also described as user-friendly.
For example, one participant reflected on the visuals and accessibility of information: “I liked
how visual it was, it was really user-friendly in the sense that all the videos played when you
clicked them. You could expand the size, you could change the volume…I thought that was
really accessible.”
130
Half of the participants also evaluated the website’s organization—how the website’s
underlying architecture contributed to the ease of use. During the think aloud, participants
described “how everything is searchable in so many different ways…by either the component of
the diet or by your grade or by video.” For example, one participant described how the ease of
navigation contributed to the access of information and potential choices for learning.
I found that the general website was attractive, it was pretty easy to navigate the tabs at
the top, and in the recipe part there were so many choices that you could go and look so
that was, I found that nice to have all those different choices to look at.
The ease of use was facilitated by the organization of the website which allowed
participants to connect their practice to relevant information in an efficient way.
3. Extent of Technology Integration
Participants evaluated the extent of technology integration. That is, the website included
a wide range of interactive technologies, including the videos which were found to be “really
helpful.” Participants could “hear how [the teacher] questions and frames the lessons and gets
the students thinking.”
Participants also commented on specific web-based features. For example, one
participant thought “it was interesting to see that there are a lot of personal views of teaching
videos” alongside the virtual tours. She continued to explain why:
I think that that is really important, that [the personal views of teaching videos] were
side-by-side with the virtual tour…I think that it certainly is the easiest to learn from
someone where there’s a similarity of style and lessons out of context don’t always
transfer very well because when the other components aren’t built into it then it won’t
work as well.
131
Another participant described how a number of features contributed to her planning and
assessment:
I feel like [the website] has everything to make it helpful for me…it has lots of pictures and
when I click on the lessons, you can print out the lesson plan, which is amazing. It breaks
down the components of the food groups and how they help. I really especially like the
assessment component which is something I’m always grappling with so it helps as well.
4. Novelty of the Technology and Uniqueness of the Learning Environment
Participants evaluated the novelty of the technology and described how the virtual
classroom tours utilized “emerging technology” and that this contributed to their motivation to
use online resources (Song & Lee, 2014, p. 517). The virtual classroom tours also contributed
to the uniqueness of the learning environment. Participants had “opportunities to explore real-
world issues through authentic learning experiences” (Song & Lee, 2014, p. 517). For instance,
while one participant reflected on her classroom virtual tour she stated: “The most appealing
thing once I recognized it was the virtual tours and the videos I would say, actually hearing the
teachers describe whatever they do and being able to see them describe it in their classroom.”
5. Potential for Learning
Participants also envisioned potential for learning by evaluating the content with respect
to their own understanding. For instance, one participant stated:
I like the way this website broke everything down. It had a well-structured lesson plan
with exactly what the students are expected to do and what we need to do in order to
facilitate the activity.
In sum, participants evaluated many aspects of the given website—participants formed
opinions about the web-based features, the organization of the website, the ease of accessing
information, and they assessed the quality and authenticity of information, as well as the
132
potential for learning. Evaluating web-based features such as these may contribute to a website
user’s ability to plan during their learning. That is, collecting, organizing, and analyzing
information can facilitate the decision-making process as teachers use a professional
development website. In addition, gaining feedback from the real user can provide meaningful
information about what works and necessary improvements.
Theme 6: Encoding Information for Future Retrieval
Encoding information for future retrieval was the second strategy participants employed
while navigating The Balanced Literacy Diet website. Participants synthesized new information
by utilizing external aids: they took notes during their navigation and also referred to
bookmarking and emailing as strategies for retaining newly learned information. Figure 4.9
presents the topics pertaining to the sixth theme.
Theme 6: Encoding Information for Future Retrieval
1. Note-Taking
2. Saving Ideas for Future Use
Figure 4.9. Topics pertaining to theme 6: Encoding information for future retrieval.
1. Note-Taking
More than half of the participants referred to note-taking as a strategy for encoding
information. Participants who took notes believed “that note taking is always a good active way
to interact with information.” One participant stated, “I like to take notes to remind myself”
while another participant explained, “if I see something that interests me I would be jotting
down little notes to remind myself…key concepts and how it was laid out so I can copy that at
some point.” Similarly, another participant stated that she “would usually take notes on the
side…jot the ideas down for [herself] to make sure [she] can revisit [the information].”
133
Participants referred to note-taking as a general strategy for encoding information while
navigating any online professional resource. For example, the following quote describes one
participant’s thought process about note taking:
The teacher [referring to a teacher in a video] seemed to have a nice comprehensive program
and she gave some nice tips off-hand that I actually wanted to write down …I do end up
opening a window to start typing down some ideas because I would want to go back to these
quickly, refer back to these quickly the next time I wanted to go through this video…or the
next time I wanted to think about some of the ideas from this so I’d have a summary for
myself.
2. Saving Ideas for Future Use
Four participants also referred to bookmarking and emailing as strategies for encoding
information while using online professional resources. For instance, one participant stated, “If
this was my computer I would have bookmarked these…just so that I could go back to them.”
Similarly, another participant “thought about emailing to [herself] because that’s usually a good
way to remind yourself of good things that you’ve done recently.”
In sum, participants in the current study encoded information during their 20-minute
website navigation by utilizing external aids, including note-taking, bookmarking, and emailing.
Many participants made references to the contribution of these external aids: for instance, the
retention and future retrieval of newly acquired information. These types of external aids can be
thought of as learning strategies and were employed by the participants (the learners) to
accomplish a specific task (to seek out information related to their teaching practice).
Furthermore, participants used these external aids in a deliberate way: participants intentionally
chose to take notes and made explicit references to the benefits of this strategy. This is
consistent with studies that report adults using deliberate strategies to enhance memory (Intons-
134
Peterson & Fournier, 1986; Schryer & Ross, 2013). Note taking, in particular has been found to
facilitate memory even when the notes are not available as retrieval cues (Intons-Peterson &
Fournier, 1986). That is, the process of taking notes in itself has a positive effect on the way in
which newly learned material is encoded. In the current study, it is possible that participants
who took notes during their website navigation may have been more likely to encode, recall, and
ultimately implement the newly learned material into their teaching practice. In addition, the
findings pertaining to theme 6 suggest that as teachers use informal online learning
environments they do so with the intention to integrate their newly formed knowledge into their
classroom practice.
Potential Outcomes Resulting from Elementary Teachers’ Website Navigation
Three themes related to the potential outcomes of the theoretical model resulted from
elementary teachers’ website navigation and emerged from the analysis, and include: Theme 7:
Reflecting on Learning; Theme 8: Continued Professional Learning; and Theme 9: Intentions
for Practice. Each theme and its relevant topics are discussed separately below.
Theme 7: Reflecting on Learning
During participants’ think aloud they reflected on their learning and their use of the
given website. Reflective statements in this study were classified through the data analysis into
two categories: (1) general feelings about the website, and (2) future use of the website. Figure
4.10 presents the topics pertaining to the seventh theme.
Theme 7: Reflecting on Learning
1. General Feelings about the Website
2. Future Use of the Website
Figure 4.10. Topics pertaining to theme 7: Reflecting on learning.
135
1. General Feelings about the Website
During their think aloud, all of the participants reflected upon their general feelings
towards The Balanced Literacy Diet website. For example, one participant perceived “almost
an overwhelming amount of information at times. It felt like [she] could explore it for hours.”
Another participant echoed this thought and also described her sense of being overwhelmed: “It
was a bit overwhelming to see the number of things that go on in the room and there are several
that I would have liked to go back and look at again.” While some participants expressed a
sense of being overwhelmed, others reflected on their sense of comfort as they used the given
website and that their comfort with the technology increased over the course of the 20-minute
task. For example, while viewing the screen recording of her navigation, one participant stated:
I’m also at this point having less panic about, you know, there’s a lot in here. Now I’m
getting a sense of what is in here…and that each component definitely has a lot, but it’s
at this point that I’m sort of going, ‘okay now I know what’s here’…as opposed to that
initial, ‘what is this and how much is in here?’
2. Future Use of the Website
In addition to their general feelings towards the website, participants also reflected on
their future use of the given website. For example, one participant reflected on features of the
website she thought she would explore during her next visit. She “probably would look at some
of the other teachers’ videos because every teacher is unique and they’re probably highlighting
different parts of their program which is pretty cool.” Similarly, another participant stated:
If I had to go back to the website I think I would have watched more of the videos, like
there was a tab that I didn’t explore, and I think that was next to the virtual tours,
something about the recipes in action…I think I would have had a look at that.
136
In sum, participants in the current study reflected on their general feelings about the
website. For some participants, this was an initial sense of feeling overwhelmed while other
participants gained a sense of comfort with the technology and ease of accessing web-based
features and tools. Participants also noted certain features and web pages that they would use in
the future. Taken together, participants’ self-reflections about their learning experience align
with Shulman and Shulman’s model of teacher learning. As previously stated, when teachers
are more conscious of their own understandings, performances, and dispositions, their capacity
to learn is enhanced (Shulman & Shulman, 2004). In the current study, participants’ reflections
about their use of and general feelings towards the website, and their performance during their
website navigation may contribute to their future use of the website—how they will narrow
down their focus to gain personally relevant pedagogical information about literacy instruction.
Theme 8: Continued Professional Learning
As teachers move forward in their career they build a repertoire of knowledge related to
pedagogy and curriculum. Their knowledge is constructed through a variety of experiences and
professional learning opportunities. In the current study, continued professional learning was
classified through the data analysis into four categories: (1) advantages of online professional
learning, (2) increased self-efficacy, (3) feeling inspired, and (4) knowledge construction.
Figure 4.11 presents the topics pertaining to the Theme 8.
Theme 8: Continued Professional Learning
1. Advantages of Online Professional Learning
2. Increased Self-Efficacy
3. Feeling Inspired
4. Knowledge Construction
Figure 4.11. Topics pertaining to theme 8: Continued professional learning.
137
1. Advantages of Online Professional Learning
All of the participants referred to the advantages of online learning, in contrast to face-
to-face workshops, packaged texts, and teacher guides. As stated by one participant: “online
resources are kind of like a one-stop shop for me because I don’t need to go anywhere
else…they offer a range of ideas for teachers.” Another participant noted the accessibility of
online resources:
[Online resources] allow me to find new ways of teaching something or new activities to
reinforce student learning that I wouldn't otherwise have access to. It is also great to see
what other teachers are doing and pick what I like and can use in my own classroom.
Practicing teachers are often under time constraints and they “don’t have that time to walk
around and to visit people’s classrooms.” When online resources and technology are easily
accessible and include real-life examples, “it makes you feel like you’re really experiencing the
lesson.”
This finding is consistent with the current study’s factors that motivate teachers to use
online resources. As previously discussed, participants’ use of online resources is largely due to
increasingly available and convenient online resources. As a potential outcome for the current
study’s model, the advantages of online resources for continued professional learning may
contribute to elementary teachers’ ongoing use of professional development websites. This
finding is also consistent with self-directedness in adult learning. Online resources are
advantageous in that they foster a self-initiated process of learning, personal autonomy, and
greater sense of learner control.
2. Increased Self-Efficacy
Teacher efficacy can be described as “teachers’ beliefs about their own capacities as
teachers” (Tschannen-Moran, Woolfolk Hoy, & Hoy, 1998). Factors that impact a teacher’s
138
efficacy include performance accomplishments, vicarious experiences, and social persuasion
(Protheroe, 2008). In the current study, vicarious experiences appeared to contribute to
participants’ increased self-efficacy. Vicarious experiences occurred as participants observed a
model teacher using an effective method of instruction. As a result, participants referred to
gains in their confidence—participants believed that they too could successfully implement a
lesson or literacy-related strategy (Protheroe, 2008). Specifically, 12 participants shared
positive reflections about their own ability to successfully employ similar instructional strategies
after they viewed videos of model teachers implementing an effective strategy. For example,
one participant stated: “I really did like exploring that teacher’s classroom…I found what she
was saying from the get-go seemed to resonate with my understanding of a literacy program.”
Similarly, another participant referred to the virtual tours as an approach to overcome feeling
isolated:
Sometimes when you are teaching, I mean I find as a teacher, once you’re actually
teaching you don’t get to spend that much time in other classrooms. This is something
that most teachers will say—that it can be kind of isolating…I know that when I see
other classrooms it’s kind of neat to go, ‘oh this teacher has an anchor chart about parts
of speech on the wall’, and I guess especially as a new teacher it can kind of affirm, oh
yeah, it does make sense that I’m doing that.
In addition, a participant described the positive impact of seeing strategies reinforced by
model teachers: “I think it’s nice to see things that you’re doing reinforced in what other people
are doing. It just keeps your confidence going…okay, I’m on the right track I’m doing the right
thing…you’re always questioning yourself.”
In sum, participants’ sense of self-efficacy appeared to be related to the success of a
model teacher they viewed in a video or virtual classroom. This finding suggests that teachers’
139
informal online learning experiences are enhanced by videos of model teachers demonstrating
effective instructional practices. These videos appear to have a positive impact on (and can
potentially change) a teacher’s sense of self-efficacy. Moreover, participants described a sense
of affirmation—a confirmation of their instructional decisions. Research that has investigated
teacher efficacy has found a positive relationship between higher levels of teacher efficacy and
classroom quality and literacy instruction (Guo, Piasta, Justice, & Kaderavek, 2010; Tschannen-
Moran & Johnson, 2011). Thus, by viewing videos of effective classroom instruction during
elementary teachers’ informal online learning experiences, teacher efficacy may increase and
contribute to the quality of classroom practice.
3. Feeling Inspired
More than half of the participants were inspired as they navigated the given website.
The Balanced Literacy Diet website “can show you things that you haven’t thought of and give
you inspiration and new ideas.” For example, one participant stated that “it’s good to get
inspired.” She went on to explain how the use of online resources for professional learning
allows her to reflect and ask:
Is there something I can form or reform or clarify? Or something that I’ve done in the
past that just has not worked at all that I’m interested in revising, like the social studies
for grade two that we’re currently mired in?
Professional development websites allow elementary teachers to look “for things that
you have seen and you want to build on and then things that you’ve never seen before.” One
participant, who identified herself as “a relatively new teacher,” recognized the impact of
discovering new ideas and “refreshing my knowledge of current practices that I can implement
in my own classroom to enhance my program and help me to become a better teacher.”
140
In the current study, the process of acquiring information via a professional development
website appeared to stimulate participants’ thinking about their individual practice—how they
could enhance their program and instructional methods. For novice teachers, the use of a
professional development website may elicit fresh ideas or clarify particular teaching
approaches. For the experienced teacher, professional development websites may reactivate
ideas and re-energize their willingness to implement new instructional strategies.
4. Knowledge Construction
As participants navigated the given website they constructed knowledge about literacy
instruction and literacy teaching strategies. For example, one participant, who “watched a video
in detail,” was “just trying to figure out how to present the concept of success criteria to the
students and how to write it out in a clear way.” Consistent with Shulman and Shulman’s model
of teacher learning, this participant began to understand the content from the website in relation
to her own practice. In addition, a participant referred to the “idea of colour coding” as an
accommodation for students with learning disabilities: “Some students I think would find it
helpful when we’re writing paragraphs or any sort of writing that can be kind of
compartmentalized to learn the form.” This participant was making sense of the information on
the website by building onto her current knowledge about classroom accommodations.
As participants navigated The Balanced Literacy Diet website, they formed concepts and
constructed ideas in relation to their individual practice. That is, they verbalized how they
would use the information presented on the website in their practice, an important element of
professional learning.
Theme 9: Intentions for Practice.
During participants’ think aloud, all of the participants shared their intentions to
incorporate newly acquired information into their teaching practice. Theme 9: Intentions for
141
Practice was classified through the data analysis into two categories: (1) adapting information,
and (2) professional goals. Figure 4.12 presents the topics pertaining to the ninth theme.
Theme 9: Intentions for Practice
1. Adapting Information
2. Professional Goals
Figure 4.12. Topics pertaining to theme 9: Intentions for practice.
1. Adapting Information
Participants shared their intentions to adapt information to their student needs. They
discussed how they would “tweak” lesson examples from the website. For instance, one
participant described how she would adapt a writing idea to her drama class: “That was
interesting, ‘All about Me Autobiographies,’ because I was starting to think that about how it
could translate into a drama activity fairly quickly.” Similarly, as participants navigated the
given website they realized “that you could combine a number of these ideas and make them
your own, see what works for you.” For example, one participant explained how a primary
level activity could be adapted to the junior level:
I’m thinking about the idea of having students have their individual cubby
holders…what would I do because I’m working with junior students, is maybe give them
a binder that they can work with…I look at these [lessons] and just know what I can do
and I have an idea of how to set things up.
More than half of the participants looked beyond the given information on The Balanced
Literacy Diet website—they extended ideas to meet their individual needs. One participant
stated, “I very rarely use anything verbatim that I find on the Internet. I usually adapt it to
something I’ve already done or I fold it into something that I’m doing.” Similarly, another
participant described an integrated activity that stemmed from a writing lesson:
142
I’m looking at text structures here because thinking about drama, there’s a lot you can do
teaching text structures through dramatic improvisation or games or exploration, and so I
decided to have a quick look and see how it’s explained.
In sum, participants in the current study made explicit references to how they would
adapt information to their own practice. This finding is consistent with Shulman and Shulman’s
model of teacher learning in that elementary teachers have a clear vision about their goals and
adapt information according to their student needs.
2. Professional Goals
Participants also described curriculum areas and aspects of their practice they believed
they could improve upon—areas they believed were necessary “to continue working on.” For
instance, one participant described how she “chose the image of the writing centre because
that’s an area that [she] would like to do better.” Similarly, another participant stated, “one area
that I’m still a little bit nervous with is doing writing, so I like to get other people’s ideas and
how they approach strategies for writing.”
Participants also referred to school and board-wide initiatives. For example, one
participant stressed how “right now there’s a lot of emphasis on anchor charts and it’s
something that I know I need more experience on, what it looks like and how I can do to model
it.” Similarly, another participant described that “at the first school that I worked at we were
involved in moderated marking throughout the whole school, from grade one to eight, with the
writing continuums.” This participant was interested in viewing “writing assessment
tools…because [she] always finds [writing assessments] tricky.”
In sum, the potential outcomes as a result of navigating The Balanced Literacy Diet
website are the “changes in cognition or behaviour resulting from engagement in activity”
(Hoekstra et al., 2009, p. 665). The primary goal of any resource is to contribute meaningful
143
information to professional practice. The findings pertaining to theme 9 are consistent with
Bakkenes and her colleagues’ findings (2010)—teachers try new practices in the context of
educational innovation and change.
General Discussion
The aim of study 2 was to utilize the virtual revisit think aloud method and two
additional data sources to examine elementary teachers’ learning experiences as they navigated
a multimedia professional development website. The findings provide a comprehensive picture
of the participants’ use of a professional development website; thorough descriptions and
explanations related to higher cognitive processes provide insights into four aspects of the
participants’ online learning experience: (1) the conditions that affect elementary teachers’ use
of a professional development website, (2) elementary teachers’ general pattern of website
navigation, (3) the navigational strategies elementary teachers employ when using a professional
development website, and (4) the potential outcomes of utilizing a professional development
website. These aspects were organized into the theoretical model for elementary teachers’
learning processes as they navigated the given website. This model offers contributions to the
literature, both for understanding teacher cognition and learning, and for informal online
professional development. Moreover, this model was developed utilizing the virtual revisit
think aloud, a method that tracked participants’ cognitive processes as they used and learned
from The Balanced Literacy Diet website; thus, this model also offers support for the virtual
revisit as an effective think aloud methodology in the context of online learning.
The general discussion section of this chapter continues with an overview of the major
findings. Next, this section presents the study limitations. Then, this section discusses the
significance of this research and possible future directions.
144
Overview of Major Findings
Based on study 2’s model for elementary teachers’ learning experience as they navigated
a professional development website, 14 major findings are summarized below. These include
conditions that affect elementary teacher’s use of a professional development website (key
conditions), strategies that elementary teachers employ while navigating a professional
development website (key navigational strategies), and potential outcomes of elementary
teachers’ use of professional development websites (key outcomes). While the term, elementary
teachers is used, the major findings are based on 15 participants from the current study.
Key Conditions:
1. Elementary teachers who utilize the Internet for their professional learning want to ensure
that professional online resources are credible and trustworthy. Specifically, elementary
teachers trust online resources that they perceive as authentic; teachers want to ensure that
the website is supported by credible organizations; and teachers value a professional
resource when it is recommended by a credible source.
2. Elementary teachers hold positive beliefs about research-based practices including best
practices in literacy instruction. In their quest for professional knowledge and instructional
strategies, elementary teachers seek information according to these beliefs.
3. Elementary teachers focus on their student needs and instructional goals as they seek
information related to their professional practice. Specifically, elementary teachers are
cognizant of their students’ needs, unique teaching context, and current and future
instructional goals.
4. Elementary teachers are motivated to use online resources for their professional learning
because these resources are convenient and accessible, and they remove time, location, and
145
situational barriers. In addition, elementary teachers are intrinsically motivated to expand
their repertoire of professional knowledge via informal online learning environments.
5. Elementary teachers vary in their use of online professional development websites:
elementary teachers have preferences for audio and visual website tools and they sometimes
multitask as they seek information related to their practice. In addition, elementary teachers
view themselves as a specific type of online learner.
6. Elementary teachers’ prior experience and professional knowledge, including the area of
their expertise, influence their navigational choices.
Key Navigational Strategies:
7. As elementary teachers navigate professional development websites they evaluate website
features and content. Elementary teachers want to ensure the applicability of the
information they are seeking to their teaching context.
8. Elementary teachers who access online resources employ encoding strategies as they use
professional development websites. Specifically, external aids (note taking, emailing, and
bookmarking) may facilitate the retention and recollection of newly learned material.
Key Potential Outcomes:
9. Elementary teachers are self-reflective during online learning. As elementary teachers use
professional development websites they reflect on their feelings towards the resource, its
content and organization, and whether they would invest in the resource over time.
10. Elementary teachers value informal online environments as a source of continued
professional development and see several advantages to this facet of learning (e.g.,
accessibility).
11. Utilizing online resources as a form of continued professional development contributes to an
increase in elementary teachers’ self-efficacy. When observing model teachers
146
implementing an effective strategy in a video or classroom virtual tour, viewers gain
confidence in their own ability to successfully employ similar strategies.
12. Utilizing professional development websites inspires elementary teachers to implement best
practices in their classroom.
13. Professional development websites contribute to elementary teachers’ knowledge
construction by providing opportunities to build upon previously acquired knowledge.
14. Elementary teachers adapt professional development to their instructional needs and
consider their professional goals when navigating online resources.
Study 2 Limitations
There are several limitations to this study that need to be considered when interpreting
the findings. The first limitation of study 2 is the spread of data collection over time. Data
collection occurred over a period of eight months and this period may have skewed the findings.
That is, the participants may have varied in their use of the website depending on the time of
year (e.g., how one teacher plans and learns at the beginning of the school year may look quite
different than how a teacher plans and learns midyear). Due to the low response rate, the eight
month period was necessary to collect data from all participants; however a shorter period of
data collection may increase the validity of the findings.
The second limitation of study 2 relates to the low response rate and the difficulty
recruiting elementary teachers to volunteer to participate during the school year. Practicing
teachers are extremely busy, and although multiple sources of participant recruitment were
employed, the response rate was low. Studies with a larger sample size that can utilize
alternative modes of recruitment, such as a greater monetary incentive to participate, will be
useful to verify the findings of this study.
147
The participants who did participate ranged in terms of their years of teaching
experience. While a range of experience contributes findings from a broader population,
experience may also influence how a teacher approaches professional development. Future
research could examine the differences in online learning experiences between expert and
novice teachers as well as between practicing and preservice teachers.
Although strategies for establishing trustworthiness were taken, the ability to generalize
qualitative findings is limited. While generalizing the findings is limited, the small sample size
does generate data that is rich and thorough, with descriptions that are thick with details about
the context, participants, and direct quotes. Therefore, the findings can guide researchers and
future work in this field of study.
A further limitation relates to reactivity which may have occurred as a result of
participants’ awareness of completing the website task and think aloud. As described in study 1,
reactivity was reduced as much as possible during the one-on-one meeting. However, reactivity
may have contributed to the verbalization of specific words related to Theme 7: Reflecting on
Learning. Some participants expressed feeling overwhelmed, stressed, anxious, and panicked in
relation to their initial use of the website. While these verbalizations were interpreted as
reflections on learning, reactivity is noted as a factor that may have influenced participants’
word choice.
Finally, it is difficult to completely eliminate researcher bias. Steps were taken to reduce
researcher bias as much as possible: staying close to the data during analysis, the application of
a well-established research approach (grounded theory), and strategies for establishing the
credibility of the findings were taken (i.e., the member check). However, this limitation must be
considered when interpreting the findings. To avoid researcher bias, future research could
involve a team of researchers with varying backgrounds, particularly during data analysis.
148
Significance and Future Directions
The significance of this study is threefold: (1) this research describes a professional
learning experience in which elementary teachers engage on a regular basis; (2) this research
identifies key conditions, strategies, and potential outcomes of elementary teachers’ use of a
professional development website; and (3) finally this research utilizes the virtual revisit think
aloud method to gain insights into the learning experiences of elementary teachers’ as they use a
multimedia professional development website.
First, this study describes how elementary teachers’ use and learn from a professional
development website—an experience in which elementary teachers often engage. Teacher
professional development is complex and multifaceted—there are multiple contexts in which
teachers’ gain professional knowledge and instructional materials. As previously mentioned,
elementary teachers are increasingly turning to the Internet for their professional learning;
therefore, it is imperative to study teacher thinking and learning in the context of informal online
learning environments. The study of teachers’ thinking in online environments can contribute to
research in teacher cognition—what teachers know, think, believe, and value. While research in
the area of teacher cognition is very extensive, research that explores elementary teachers’
thinking during informal online professional learning is limited. Thus, the findings from this
study provide a representation of teachers’ learning experiences in one domain of teacher
professional development. In addition, this representation can benefit educational website
designers and policy makers by helping them anticipate and meet the needs of teachers in their
search for high-quality professional resources.
Secondly, this research identifies key conditions, navigational strategies, and potential
outcomes related to elementary teachers’ use of a professional development website. Research
in the field of adult learning is vast and, as discussed in the introduction, models of the adult
149
learner as well as theories about how adults learn can support the design and implementation of
professional development across many domains. The findings of study 2 portray an in-depth
representation of teacher learning within informal online environments. How elementary
teachers use and learn from informal professional development websites is a relatively under-
examined area of adult learning and teacher professional development. However, identifying
specific aspects of teachers’ informal online learning experiences can contribute to a holistic
understanding of teacher professional development; that teacher learning is complex and
multifaceted.
Finally, this study utilizes the virtual revisit think aloud method to explore elementary
teachers’ informal online learning experiences. As previously discussed, the virtual revisit think
aloud provides direct data about the reasoning process: combining the retrospective think aloud
and screen capture recording allows participants to reason about their decisions; they describe
their choices and why they made them. The findings related to the virtual revisit further
substantiate study 1 findings: the virtual revisit appears to be an effective approach for exploring
the use of Internet-based resources by elementary teachers. In study 2, participants were able to
access their interactive decision making immediately after viewing a recording of their
navigational experience. They viewed details of the website and offered reasons for their web-
based behavoiurs—participants clarified their choices, explained their decisions, and provided
details about their navigation. Access to participants’ cognitive processes was possible via the
virtual revisit think aloud, and contributed to a theoretical model for elementary teachers’
learning experiences as they navigated a professional development literacy website.
The virtual revisit allowed participants to focus on what they were doing at a particular
time while freeing them from having to complete the task and simultaneously talk about it.
Furthermore, the limitations of the concurrent and retrospective techniques may have been
150
avoided with the virtual revisit think aloud method. The virtual revisit think aloud allowed
participants to recall their decisions and why they made them. Future research that investigates
the virtual revisit think aloud methodology for examining teacher cognition within a variety of
informal and formal online environments will contribute to the validity of this methodology. In
addition, the virtual revisit think aloud could be applied across a range of age groups and
domains in education. If this methodology can provide access into elementary teachers’
thinking as they utilize a professional development website, the application of the virtual revisit
to any online learning experience is possible.
151
CHAPTER 5
General Conclusions
The aim of this dissertation was twofold: (1) to compare three think aloud methods—
concurrent, retrospective and virtual revisit—for examining elementary teachers’ cognitive
processes as they use a multimedia professional development website; and (2) to gain an in-
depth understanding of the teachers’ learning experiences from the virtual revisit condition as
they navigate a professional development website. The findings of study 1 suggest that the there
are benefits and limitations to each type of think aloud method. In addition, findings of study 1
indicate that the virtual revisit think aloud methodology can overcome the limitations of the
concurrent and retrospective think alouds. In contrast to the concurrent think aloud, the virtual
revisit method produces rich and thorough verbalizations related to teacher planning, reasoning,
and reflection. In contrast to the retrospective think aloud, the virtual revisit method produces
details concerning the entire task—little to no information is omitted, as in the retrospective
procedure. Thus, in addressing the research questions related to study 1, this exploratory study
suggests that there are differences in verbalizations between the concurrent and virtual revisit
think aloud as well as between the concurrent and retrospective think aloud methods. In
addition, differences in the think aloud task and comfort level with thinking aloud across the
think aloud methods.
Given the findings of study 1, data from the virtual revisit condition was further
examined in a second study using a grounded theory approach to analysis. The findings of
study 2 provided a theoretical model of elementary teachers’ learning experiences as they used a
multimedia professional development website. The nine themes that emerged as a result of the
analysis present three conditions that affect elementary teachers’ use of a professional
development website, the general pattern of website navigation, two navigational strategies
elementary teachers employ when using a professional development website, and three potential
152
outcomes of utilizing a professional development website. As discussed in the previous chapter,
this model offers contributions to the literature, both for understanding teacher cognition and
learning, and for informal online professional development.
The overall findings of this research suggest that elementary teachers actively construct
pedagogical knowledge and instructional teaching methods as they use online learning
environments. To capture their thought processes as accurately as possible, a methodology like
the virtual revisit think aloud must be employed. As elementary teachers increasingly turn to
the Internet for their professional learning, it is essential to examine how teachers use
professional websites and web-based technologies. By utilizing the virtual revisit method,
researchers can gain direct insights into elementary teachers’ cognitive processes and learning
experiences (conditions, navigational strategies, potential outcomes) as they use and learn from
professional development websites. This information can contribute to the ongoing
development of high quality online learning environments. If researchers provide feedback to
educational website developers based on the methodology used in the present study,
improvements can be made to the design and content of their sites. Similarly, findings may
benefit policy makers and administrators of professional development by providing insights
concerning elementary teachers’ use of informal online resources as an aspect of their
professional development.
153
References
Agostinho, S. (2005). Naturalistic inquiry in e-learning research. International Journal of
Qualitative Methods, 4(1), 1-13.
Anderson, J. R. (1995). Cognitive Psychology and Its Implications. (4th edition). New York: W.
H. Freeman and Company.
Andreassen, R., & Braten, I. (2013). Teachers’ source evaluation self-efficacy predicts their use
of relevant source features when evaluating the trustworthiness of web sources on
special education. British Journal of Educational Technology, 44(5), 821-836.
Aranyi, G., van Schaik, P., & Barker, P. (2012). Using think-aloud and psychometrics to explore
users’ experience with a news website. Interacting with Computers, 24, 69-77.
Armstrong, D. (2011). Students’ perceptions of online learning and instructional tools: A
qualitative study of undergraduate students’ use of online tools. The Turkish Online
Journal of Educational Technology, 10(3), 222-226.
Bakkenes, I., Vermunt, J.D., Wubbels, T. (2010). Teacher learning in the context of educational
innovation: Learning activities and learning outcomes of experienced teachers. Learning
and Instruction 20, 533-548.
Barak, M. (2010). Motivating self-regulated learning in technology education. International
Journal of Technology and Design Education, 20(4), 381-401.
Barzilai, S. & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online
sources. Cognition and Instruction, 30(1), 39-85.
Boren, M.T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE
Transactions on Professional Communication, 43(3), 261-278.
Borg, S. (2006). Teacher cognition and language education: Research and practice. London:
Continuum.
154
Borko, H. (2004). Professional development and teacher learning: Mapping the Terrain.
Educational Researcher, 33 (8), 3-15.
Borko, H., Livingston, C., & Shavelson, R.J. (1990). Teachers’ thinking about instruction.
Remedial and Special Education, 11(6), 40-49.
Branch, J. (2006). Using think alouds, think afters, and think togethers to research adolescents’
inquiry experiences. Alberta Journal of Educational Research, 52(3), 148-159.
Brockett, R.G., & Hiemstra, R.H. (1985). Bridging the theory-practice gap in self-directed
learning. New Directions for Continuing Education, 25, 31-40.
Callanan, M., Cervantes, C., & Loomis, M. (2011). Informal learning. Wiley Interdisciplinary
Review: Cognitive Science, 2(6), 646-655.
Caffarella, R. (1993). Self-directed learning. New Directions for Adult and Continuing
Education 57, 25-35.
Callahan, C., Saye, J., & Brush, T. (2013). Designing web-based educative curriculum
materials for the social studies. Contemporary Issues in Technology and Teacher
Education, 13(2), 126-155.
Candy, P. (1988). Key issues for research in self-directed learning. Studies in Continuing
Education, 10(2), 104-124.
Carcary, M (2009). The research audit trail: Enhancing trustworthiness in qualitative inquiry.
Electronic Journal of Business Research Methods, 7(1), 11-24.
Cercone, K. (2008). Characteristics of adult learners with implications for online learning
design. Association for the Advancement of Computing in Education Journal, 16(2),
137-159.
Cervetti, G.N., Kulikowich, J., Drummond, M., & Billman, A. (2012). Influence of educative
155
curriculum materials on teachers’ learning and teaching. Paper presented at the annual
meeting of the American Educational Research Association, Vancouver, B.B. Retrieved
August 8, 2014 from
http://www.sandi.net/cms/lib/CA01001235/Centricity/Domain/121/R_R/DistResearch/R
esultReports/585_Cervetti_G_et_al_UCBerkeley_2012B.pdf
Charalambousa, K., & Ioannou, I. (2011). The attitudes and opinions of Cypriot primary
teachers about the use of the Internet for their professional development and as an
educational tool. Learning, Media and Technology, 33(1), 45-57.
Chametzky, B. (2014). Andragogy and engagement in online learning: Tenets and solutions.
Creative Education, 5, 813-821.
Charmaz, K. (2014). Constructing grounded theory: A practical guide through qualitative
Analysis (2nd
ed.). Thousand Oaks, CA: SAGE Publications.
Clark, C.M., & Peterson, P.L. (1986). Teachers' thought processes. In M.C. Wittrock, (Ed.),
Handbook Of Research Teaching (3rd
ed.) (pp.255-296). New York: MacMillan
Publishing Company.
Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A
technical communication approach. IEEE Transactions on Professional Communication,
53(3), 202-215.
Corbin, J.M., & Strauss, A.L. (2008). Basics of qualitative research: Techniques and
procedures for developing grounded theory (3rd
ed.). Thousand Oaks, CA: SAGE
Publications.
Creswell, J.W., & Piano Clark, V.L. (2007). Designing and conducting mixed methods research.
Thousand Oaks: Sage Publication.
Cunningham, A.E., Zibulsky, J., Stanovich, K.E., & Stanovich, P.J. (2009). How teachers would
156
spend their time teaching language arts. Journal of Learning Disabilities, 42(5), 418-
430.
Damico, J., & Balidon, M. (2007). Examining ways readers engage with websites during think-
aloud sessions. Journal of Adolescent and Adult Literacy, 51(3), 254-263.
Darling-Hammond, L., & Richardson, N. (2009). Teacher learning: What matters? Educational
Leadership, 46-53.
Davis, E.A., & Krajcik, J.S. (2005). Designing educative curriculum materials to promote
teacher learning. Educational Researcher, 34(3), 3-14.
Davis, E.A., Palincsar, A.S., Arias, A.M., Bismack, A.S., Marulis, L.M., & Iwashyna, S.K.
(2014). Designing educative curriculum materials: A theoretically and empirically driven
process. Harvard Educational Review, 84(1), 24-52.
Deci, E. L., & Ryan, R. M. (2008). Self-determination theory: A macrotheory of human
motivation, development, and health. Canadian Psychology, 49(3), 182-185.
Desimone, L. M. (2009). Improving impact studies of teachers’ professional development:
Toward better conceptualizations and measures. Educational Research, 38(3), 181-199.
de Vries, S., van de Grift, W., & Jansen, E. (2013). Teachers’ beliefs and continuing
professional development. Journal of Educational Administration, 51(2), 213-231.
de Vries, S., van de Grift, W., & Jansen, E. (2014). How teachers’ beliefs about learning and
teaching related to their continuing professional development. Teachers and Teaching:
Theory and Practice, 20(3), 338-357.
Dillon, A. (2001). Beyond usability: process, outcome and affect in human-computer
interactions. Canadian Journal of Library and Information Science, 26(4), 57-69.
Downer, J.T., Pianta, R.C., Fan, X., Hamre, B.K., Mashburn, A., & Justice, L. (2012). Effecits
157
of web-mediated teacher professional development on the language and literacy skills of
children enrolled in prekindergarten programs. NHSA Dialog, 14(4), 189-212.
Duncan-Howell, J. (2010). Teachers making connections: Online communities as a source of
professional learning. British Journal of Educational Technology, 41(2), 324-340.
Ebbinghaus, H. (1964). Memory: A Contribution to Experimental Psychology. New York:
Dover.
Ericsson, K. A., & Moxley, J. H. (2011). Thinking aloud protocols: Concurrent verbalizations of
thinking during performance tasks involving decision making. In Schulte-Mecklenbeck,
M., Kuhberger, A., & Ranyard, R. (Eds.), The Handbook of Process Tracing Methods
for Decision Research: A Critical Review and User’s Guide. (p. 89-114). New York,
NY: Psychology Press.
Ericsson, K. A., & Simon H. (1984). Protocol analysis: Verbal reports as data. Cambridge,
MA: MIT Press.
Ericsson, K. A., & Simon, H. (1993). Protocol analysis: Verbal reports as data (Rev. ed.).
Cambridge, MA: MIT Press.
Ericsson, K. (2002). Towards a procedure for eliciting verbal expression of non-verbal
experience without reactivity: Interpreting the verbal overshadowing effect within the
theoretical framework for protocol analysis. Applied Cognitive Psychology, 16, 981-987.
Ericsson, K. (2003). Valid and non-reactive verbalization of thoughts during performance of
tasks: Towards a solution to the central problems of introspection as a source of
scientific data. Journal of Consciousness Studies, 10(9-10), 1-18.
Field, A. O. (2009). Discovering statistics using SPSS. London, England: SAGE.
Fishman, B., Konstantopoulos, S., Kubitskey, B.W., Vath, R., Park, G., Johnson, H., & Edelson,
158
D.C., 2013. Comparing the impact of online and face-to-face professional development
in the context of curriculum implementation. Journal of Teacher Education, 64(5), 426-
438.
Fourie, I., & Bothma, T. (2007). Information seeking: An overview of web tracking and the
criteria for tracking softeware. Aslib Proceedings, 59 (3), 264-284.
Gaissmaier, W., Fific, M., & Rieskamp, J. (2010). Analyzing response times to understand
decision processes. In M. Schulte-Mecklenbeck, A. Kuhberger, & R. Ranyard (Eds.),
The Handbook of Process Tracing Methods for Decision Research: A Critical Review
and User’s Guide. (p. 89-114). New York, NY: Psychology Press.
Gambrell, L.B., Malloy, J.A., & Anders Mazzoni, S. (1999). Evidence-based best practices in
comprehensive literacy instruction. In Mandel Marrow, L. & Gambrell, L.B. (Eds.), Best
practices in literacy instruction (11-36). New York: The Guilford Press.
Garner, R., & Gillingham, (1991). Topic knowledge, cognitive interest, and text recall. The
Journal of Experimental Education, 59(4), 310-319.
Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory: Strategies for
qualitative research. Chicago: Aldine Publishing Co.
Glesne, C. (1999). Becoming qualitative researchers: An introduction (2nd ed.). New York:
Longman.
Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The
Qualitative Report, 8(4), 597-607.
Gorozidis, G., & Papaioannour, A.G. (2014). Teachers’ motivation to participate in training and
to implement innovations. Teaching and Teacher Education, 39, 1-11.
Guan, Z., Lee, S., Cuddihy, E., & Ramey. J. (2006). The validity of the stimulated retrospective
think-aloud method as measured by eye tracking. CHI, April 22-27.
159
Guskey, T.R. (2002). Professional development and teacher change. Teachers and Teaching:
Theory and Practice, 8(4), 381-391.
Guo, Y., Piasta, S.B., Justice, L.M., & Kaderavek, J.N. (2010). Relations among preschool
teachers’ self-efficacy, classroom quality, and children’s language and literacy gains.
Teaching and Teacher Education, 26, 1094-1103.
Hartree, A. (1984). Malcolm Knowles’ theory of andragogy: A critique. International Journal of
Lifelong Education, 3(3), 203-210.
Haymore Sandholtz, J. (2002). Inservice training or professional development: Contrasting
opportunities in a school/university partnership. Teaching and Teacher Education, 18,
815-830.
Hoekstra, A., Brekelmans, M., Beijaard, D., & Korthagen, F. (2009). Experienced teachers’
informal learning: Learning activities and changes in behaviour and cognition. Teacher
and Teacher Education, 25, 663-673.
Hogan, T., Rabinowitz, M., & Craven, J.A. (2003). Representation in teaching: Inferences from
research of expert and novice teachers. Educational Psychologist, 38(4), 235-247.
Horz, H., & Schnotz, W. (2010). Cognitive load in learning with multiple representations. In
Plass, J.L., Moreno, R., & Brunken, R. (Ed.), Cognitive load theory (229-252). New
York: Cambridge University Press.
Hur, J. W., & Brush, T. A. (2009). Teacher participation in online communities: Why do
teachers want to participate in self-generated online communities of K-12 teachers?
Journal of Research on Technology in Education, 41(3), 279-303.
Iding, M.K., Crosby, M.E., Auernheimer, B., & Klemm, E.B. (2009). Website credibility: Why
do people believe what they believe? Instructional Science, 37, 43-63.
Intons-Peterson, M.J., & Fournier, J. (1986). External and internal memory aids: When and how
160
often do we use them? Journal of Experimental Psychology, 115(3), 267-280.
Jamieson, B. (2007, June). Male presence in teaching continues to decline. Professionally
Speaking. Retrieved from http://professionallyspeaking.oct.ca/june_2007/
male_teachers.asp
Jaspers, M. W. (2009). A comparison of usability methods for testing interactive health
technologies: Methodological aspects and empirical evidence. International Journal of
Medical Informatics, 78, 340-353.
Jaspers, M., Steen, T., van den Bos, C., & Geenen, M. (2004). The think aloud method: A guide
to user interface design. International Journal of Medical Infomatics, 73, 781-795.
Jones, M., & Dexter, S. (2014). How teachers learn: The roles of formal, informal, and
independent learning. Educational Technology Research Development, 62, 367-384.
Jurasaite-Harbison, E., & Rex, L.A. (2010). School cultures as contexts for informal teacher
learning. Teaching and Teacher Education, 26, 267-277.
Kansanen, P., Tirri, K., Meri, M., Krokfors, L., Husu, J., & Jyrhama, Riitta. (2000). Teachers’
pedagogical thinking: Theoretical landscapes, practical challenges. New York: NY, Peter
Lange Publishing, Inc.
Kanuka, H., & Nocente, N. (2003). Exploring the effects of personality type on perceived
satisfaction with web-based learning in continuing professional development. Distance
Education, 24(2), 227-244.
Kao, C. P., Wu, Y. T., & Tsai, C. C. (2011). Elementary school teachers motivation toward
web-based professional development, and the relationship with Internet self-efficacy and
belief about web-based learning. Teacher and Teacher Education, 27, 406-415.
Kennedy, E. (2010). Improving literacy achievement in a high-poverty school: Empowering
classroom teachers through professional development. Reading Research Quarterly,
161
45(4), 384-387.
Knowles, M. (1975). Self-Directed Learning: A Guide for Learners and Teachers, New York,
NY: Association Press.
Knowles, M. S. (1977). The modern practice of adult education: Andragogy versus pedagogy
(8th ed.). New York: Association Press.
Knowles, M. S. (1980). My farewell address...andragogy: no panacea, no ideology. Training
and Development Journal, 48-50.
Knowles, M.S., Holton, E.F., & Swanson, R.A. (2011). The adult learner: The definitive
classic in adult education and human resource development (7th
Ed.). New York, NY:
Elsevier Inc.
Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol
analysis. American Journal of Psychology, 113 (3), 387-404.
Krathwohl, D. R. (2002). A revision of Bloom’s Taxonomy: An overview. Theory into Practice,
41(4), 212-218.
Laberge, J.C., & Scialfa, C.T. (2005). Predictors of web navigation performance in a life span
sample of adults. The Journal of the Human Factors and Ergonomics Society, 47(2),
289-302.
Leech, N.L., & Onwuegbuzie, A.J. (2007). An array of qualitative data analysis tools: A call for
data analysis triangulation. School Psychology Quarterly, 22(4), 557-584.
Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory into Practice, 32(3),
131-137.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Livingston, C., & Borko, H. (1989). Expert-novice differences in teaching: A cognitive analysis
and implications for teacher education. Journal of Teacher Education, 40(4), 36-42.
162
Lohman, M. C. (2006). Factors influencing teachers’ engagement in informal learning activities.
Journal of Workplace Learning, 18(3), 141-156.
Lomard, M., Snyder-Duch. J., & Campanella Bracken, C. (2004). Practical resources for
assessing and reporting intercoder reliability in content analysis research projects.
Retrieved from http://astro.temple.edu/~lombard/reliability
Loyens, S.M., Magda, J., & Rikers, R. (2008). Self-directed learning in problem-based learning
and its relationship with self-regulated learning. Educational Psychological Review, 20,
411-427.
Maass, K. (2011). How can teachers’ beliefs affect their professional development? The
International Journal of Mathematics Edcuation, 43(4), 573-586.
Mastin, L. (2010). The human memory: Memory encoding. Retrieved from
http://www.human-memory.net/processes_encoding.html
Masuda, A.M., Ebersole, M.M., & Barrett, D. (2013). A qualitative inquiry: Teachers attitudes
and willingness to engage in professional development experiences at different career
stages. Educational Research, 6-14.
Mayer, R. (2002). Cognitive theory and the design of multimedia instruction: An example of the
two-way street between cognition and instruction. New Directions for Teaching and
Learning, 2002(89), 55-71.
McDonald, L. (2012). Teacher learning, transfer and motivation: facilitating improved outcomes
for students. Journal of Educational Sciences & Psychology, 2(2), 15-24.
McDonald, S., Edwards, H., & Zhao, T. (2012). Exploring think-alouds in usability testing: An
international survey. IEEE Transactions on Professional Communication, 55 (1), 2-19.
McLeod, S. A. (2007). Cognitive Approach in Psychology - Simply Psychology. Retrieved from
http://www.simplypsychology.org/cognitive.html
163
McLoughlin, D., & Mynard, J. (2009). An analysis of higher order thinking in online
discussions. Innovations in Education and Teaching Internationally, 46(2), 147-160.
Mercier, J., Girard, C., Brodeur, M., & Laplante, L. (2010). Collaborative and individual
learning in teacher: A trajectory to expertise in pedagogical reasoning. New York:
Nova Science Publishers Inc.
Merriam, S. (2001). Andragogy and self-directed learning: Pillars of adult learning theory. New
Directions for Adult and Continuing Education, 89, 3-13.
Moos, D. (2014). Setting the stage for the metacognition during hypermedia learning: What
motivation constructs matter? Computers & Education 70, 128-137.
Mutlu Bayraktar, D., & Altun, A. (2014). The effect of multimedia design types on learners’
recall performances with varying short term memory spans. Multimedia Tools and
Applications, 71(3), 1201-1213.
National Early Literacy Panel (2008). Developing early literacy: Report of the national early
literacy panel. National Institute for Literacy.
National Reading Panel (2000). Teaching children to read: An evidence-based assessment of the
scientific research literature on reading and its implications for reading instruction.
Reports of the subgroups. Washington, D.C.: National Institute of Child Health and
Human Development.
Nielsen, J., Clemmensen, T., & Yssing, C. (2002). Getting access to what goes on in people’s
heads?-Reflections on the think-aloud technique. NordiCHI, October 19-23.
Nir, A.E., & Bogler, R. (2008). The antecedents of teacher satisfaction with professional
development programs. Teaching and Teacher Education, 24, 377-386.
NVivo, NVivo qualitative data analysis software; QSR International Pty Ltd. Version 10, 2012.
Olmsted-Hawala, E., Murphy, E., & Hawala, S. (2010). Think-aloud protocols: Analyzing three
164
different think-aloud protocols with counts of verbalized frustrations in a usability study
of an information-rich website. IEEE International Professional Communication
Conference.
Pearson, J., Pearson, A., & Green, D. (2007). Determining the importance of key criteria in web
usability. Management Research News, 30(11), 816-828.
Page K.L., Robson, M.J., & Uncles, M.D. (2012). Perceptions of web knowledge and usability:
When sex and experience matter. International Journal of Human-Computer Studies,
70(12), 907-919.
Paas, F.G., & Van Merrienboer, J. (1994). Variability of worked examples and transfer of
geometrical problem-solving skills: A cognitive-load approach. Journal of Educational
Psychology, 86(1), 122-133.
Ping Lim, C. (2004). Engaging learners in online learning environments. Tech Trends, 48(4),
16-23.
Pitsoe, V.J., & Maila, W.M. (2012). Towards constructivist teacher professional development.
Journal of Social Sciences, 8(3), 318-324.
Podhajski, B., Mather, N., Nathan, A., & Sammons, J. (2009). Professional development in
scientifically based reading instruction: Teacher knowledge and reading outcomes.
Journal of Learning Disabilities, 42(5), 403-417.
Poland, B. D. (1995). Transcription quality as an aspect of rigor in qualitative research.
Qualitative Inquiry, 1(3), 290-310.
Ponterotto, J.G. (2006). Brief note on the origins, evaluation, and meaning of the qualitative
research concept ‘thick descriptions’. The Qualitative Report, 11(3), 538-549.
Ponti, M. (2014). Self-directed learning and guidance in non-formal open courses. Learning,
Media and Technology, 39(2), 154-168.
165
Porche, M.V., Pallante, D.H., & Snow, C.E. (2012). Professional development for reading
achievement. The Elementary School Journal, 112(4), 649-671.
Postholm, M.B. (2012). Teachers’ professional development: A theoretical review. Educational
Research, 54(4), 405-429.
Pratt, D.D. (1993). Andragogy after twenty-five years. New Directions for Adult and Continuing
Education, 57, 15-23.
Pressley, M., Mohan, L., Raphael, L.M., & Fingeret, L. (2007). How does Bennett Woods
Elementary School produce such high reading and writing achievement? Journal of
Educational Psychology, 99(2), 221-240.
Protheroe, N. (2008). Teacher efficacy: What is it and does it matter? Principal, 87(5), 42.
Rachel, J.R. (2002). Andragogy’s detectives: A critique of the present and a proposal for the
future. Adult Education Quarterly, 52(3), 210-227.
Ranyard, R., & Svenson, O. (2010). Verbal data and decision process analysis. In M. Schulte-
Mecklenbeck, A. Kuhberger, & R. Ranyard (Eds.), The Handbook of Process Tracing
Methods for Decision Research: A Critical Review and User’s Guide. (p. 89-114). New
York, NY: Psychology Press.
Recker, M., Walker, A., Giersch, S., Mao, X., Halioris, S., Palmer, B., Johnson, D., Leary, H., &
Robertshaw, M.B. (2007). A study of teachers’ use of online learning resources to design
classroom activities. New Review of Hypermedia and Multimedia, 13(2), 117-134.
Reeves, T.D., & Pedulla, J.J. (2011). Predictors of teacher satisfaction with online professional
development: Evidence form the USA’s e-Learning for Educators initiative. Professional
Development in Education, 37(4), 591-611.
Rich, P., & Hannafin, M. (2008). Capturing and assessing evidence of study teacher inquiry: A
case study. Teaching and Teacher Education, 24(6), 1426-1440.
166
Rosenzweig, C., Krawec, J., & Montague, M. (2011). Metacognitive strategy use of eighth-
grade students with and without learning disabilities during mathematical problem
solving: A think-aloud analysis. Journal of Learning Disabilities, 44(6), 508-520.
Rowan, M., & Huston, P. (1997). Qualitative research articles: Information for authors and peer
reviewers. Canadian Medical Association, 157(10), 1442-1446.
Ryan, R.M., & Deci, E.L. (2000). Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. American Psychologist, 55(1), 68-78.
Schellenbach-Zell, J., & Grasel, C. (2010). Teacher motivation for participating in school
innovations—supporting factors. Journal for Educational Research Online, 2(2), 34-54.
Shenton, A.K. (2004). Strategies for ensuring trustworthiness in qualitative research projects.
Education for Information, 22, 63-75.
Schryer, E., & Ross, M. (2013). The use and benefits of external memory aids in older and
younger adults. Applied Cognitive Psychology, 27, 663-671.
Shulman, L.S., & Shulman, J.H. (2004). How and what teachers learn: A shifting perspective.
Journal of Curriculum Studies, 36(2), 257-271.
Song, D., & Lee, J. (2014). Has web 2.0 revitalized informal learning? The relationship between
web 2.0 and informal learning. Journal of Computer Assisted Learning, 30, 511-533.
Song, L., & Hill, J.R. (2007). A conceptual model for understanding self-directed learning in
online environments. Journal of Interactive Online Learning, 6(1), 27-41.
Tan, G. W., & Wei, K. K. (2006). An empirical study of web browsing behaviour: Towards an
effective website design. Electronic Commerce Research and Applications, 5, 261-271.
Taylor, B., & Kroth, M. (2009). A single conversation with a wise man is better than ten years
of study: A model for testing methodologies for pedagogy or andragogy. The Journal of
Scholarship of Teaching and Learning, 9, 42-56.
167
Thomas, D. (2006). A general inductive approach for analyzing qualitative data. American
Journal of Evaluation, 27(2), 237-246.
Tilley, S. A. (2003). “Challenging” research practices: Turning a critical lens on the work of
transcription. Qualitative Inquiry, 9(5). 750-773.
Trotter, Y. (2006). Adult learning theories: Impacting professional development programs. The
Delta Kappa Gamma Bulleti, 72 (2), 8-13.
Tschannen-Moran, M., & Johnson, D. (2011). Exploring literacy teachers’ self-efficacy beliefs:
Potential sources at play. Teaching and Teacher Education, 27, 751-761.
Tschannen-Moran, M., Woolfolk Hoy, A., & Hoy, W.K. (1998). Teacher efficacy: Its meaning
and measure. Review of Educational Research, 68(2), 202-248.
UNESCO. (2009). Open educational resources: Conversations in cyberspace. S. D’Antoni, &
C. Savage, Eds. Paris: UNESCO.
Uppal, T. (2011). Internet use for lesson preparation by pre-service teachers: A mixed methods
study. (Unpublished Master’s thesis.) OISE, University of Toronto, Toronto, Canada.
van den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2003). Retrospective vs. concurrent
think-aloud protocols: Testing the usability of an online library catalogue. Behaviour &
Information Technology, 22(5), 339-52.
van Gog, T., Kester, L., Nievelstein, F., Giesbers, B., & Paas, F. (2009). Uncovering cognitive
processes: Different techniques that can contribute to cognitive load research and
instruction. Computers in Human Behavior, 25, 325-331.
van Gog, T., Paas, F., van Marrienboer, J., & Witte, P. (2005). Uncovering the problem-solving
process: Cued retrospective reporting versus concurrent and retrospective reporting.
Journal of Experimental Psychology: Applied, 11 (4), 237-244.
168
van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. (1994). The think aloud method: A
practical guide to modelling cognitive processes. London: Academic Press.
Vermunt, J.D., & Endedijk, M.D. (2011). Patterns in teacher learning in different phases of the
professional career. Learning and Individual Differences 21, 294-302.
Warren, L.L. (2000). Teacher planning: A literature review. Educational Research Quarterly,
24(2), 37.
Weber, K., Corrigan, M., Fornash, B., & Neupauer, N. (2003). The effect of interest on recall:
An experiment. Communication Research Reports, 20(2), 116-123.
Westerman, D.A. (1991) Expert and novice teacher decision making. Journal of Teacher
Education, 42(4), 292-305.
Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., & Hemmerich, J.A. (2009).
Source evaluation, comprehension, and learning in Internet science inquiry tasks.
American Educational Research Journal, 46(4), 1060-1106.
Wing-mui So, W. (1997). A study of teacher cognition in planning elementary science lessons.
Research in Science Education, 27(1), 71-86.
Zhao, T., & McDonald, S. (2010). Keep talking: An analysis of participant utterances gathered
using two concurrent think-aloud methods. Proceedings: NordiCHI, 16-20.
169
Appendices
Appendix A-1
Sample Pages of The Balanced Literacy Diet Website
Food Group Page Example
Recipe Finder Page Example
170
Recipe Example
Virtual Tour Example
172
Appendix A-2
Invitation to Participate Script
My name is Pamela Beach. I am currently a PhD student at the Ontario Institute for Studies in
Education at the University of Toronto. I would like to invite you to be a part of a research
project that I am completing involving the use of educational websites by elementary teachers. I
am conducting this project for my dissertation and to learn more about how elementary teachers
use an educational website to assist in professional development in literacy instruction.
If you decide to participate, you will first be asked to complete an online questionnaire. The
website link to the questionnaire will be emailed to you and should take about 15 minutes to
complete. You will then be contacted to arrange a one-on-one meeting session.
The one-on-one session will take about one hour and can occur at a time that is most convenient
to you. During the session, you will be asked to navigate through a professional development
website and think aloud about your navigational experience. A list of questions will then be
asked concerning your experience of exploring the website as well as your general feelings
towards thinking aloud.
Your involvement would be greatly appreciated and upon completion of the tasks you would
receive a gift certificate from Indigo Book Store. Please let me know if have any questions
regarding this project. If you do decide to participate, please complete a consent form, place it in
the provided envelop and leave it in the designated box. I will return at the end of this class to
collect the box.
173
Appendix A-3
Information Letter and Consent Form
Dear Participant,
My name is Pamela Beach. I am currently a PhD student at the Ontario Institute for Studies in
Education at the University of Toronto. I would like to invite you to be a part of a research
project that I am completing involving the use of educational websites by elementary teachers. I
am conducting this project to learn more about how elementary teachers use an educational
website to assist in professional development in literacy instruction and what elementary
teachers think about while navigating online resources. I am asking you to take part in this
research because I believe that your experiences and opinions are valuable information.
Description of the Project
If you take part in this research, you will be asked to complete an online questionnaire. The
website link to the questionnaire will be emailed to you and should take about 15 minutes to
complete. You will then be contacted to arrange a one-on-one meeting session.
The one-on-one session will take about one hour and can occur at a time that is most convenient
to you. During the session, you will be asked to navigate through a professional development
website and think aloud about your navigational experience. A list of questions will then be
asked concerning your experience of exploring the website as well as your general feelings
towards thinking aloud. You will be asked to complete a second consent form at this time.
Upon completion of the tasks, you will receive a gift certificate to a book store.
Benefits
The main benefit of participating is that you will help us learn more about teachers and the use
of an innovative educational website. This will allow us to evaluate this website and its features
as online professional resources. We will also begin to understand how elementary teachers use
professional development websites for their teaching practice.
Potential Harms and Withdrawal There are no harms associated with taking part in this research. If at any time you feel that you
do not want to answer a question, you can tell the researcher, and talk about it. You may also
say that you want to stop, skip a question, or that you need a break and want to continue at some
other time. Also, if you say that you will take part in the research and then change your mind,
this is okay. You can decide at any time to stop taking part in the research. The only
consequence is that we would not have the information we need to write the report that we
described above.
174
Confidentiality
Your answers will only be used for research. No information that reveals your identity will be
released without consent, unless required by law. I will not put your name or your personal
information on any of the activities we do together, or include them in anything that I write or
present describing the research results. These will be kept in locked filing cabinets in our
research lab. Electronic data will be stored on a secure server. We will use a code on these files
instead of your name and all of the files will be encrypted.
Consent-Participation In This Study Is Your Choice
It is completely up to you to decide if you would like to be part of this research. If you are
interested to participate, you will be asked to sign the consent form. As a participant, you may
skip any questions that you do not want to answer, ask for a break and/or leave the study at any
time. I will be happy to answer any of your questions. If you would like to talk about taking
part in this study with someone who is not involved in it, you may call the Ethics Review Office
at the University of Toronto.
Access to Results
A summary of the overall results of this study can be mailed/e-mailed to you about 1 year after
the study. If you would like a copy of the results, please place a checkmark in the box .
As well, you can always contact Dr. Willows or me at any time with questions that you may
have about the study or its results.
Thank you,
175
Consent:
“I acknowledge that the research procedures described above have been explained to me and
that any questions that I have asked have been answered to my satisfaction. As well, the
potential harms and discomforts have been explained to me and I also understand the benefits of
participating in the research study. I know that I may ask now, or in the future, any questions
that I have about the study. I have been assured that no information will be released or printed
that would disclose my identity without my permission, unless required by law. I understand
that I will receive a copy of this signed consent. I understand that participation is voluntary and
I can withdraw at any time.”
I hereby consent to take part in this research.
Name (please print)
Email (please use the address to which you
would like the survey to be sent)
Signature
Date
“I agree that the information gathered in this study can be used for future data analysis as long
as my personal information (i.e. name) is not used.”
Signature
If you are interested in participating in this study, please fill out the attached consent form
and return it in the envelope provided to the designated box.
176
Appendix A-4
Recruitment Poster
Department of Applied Psychology and Human Development
For more information please contact:
Are you a practicing elementary school teacher?
Do you currently teach at the primary or junior level?
If so, you are invited to participate in a research
study on the use of educational websites by
elementary teachers.
Participation will involve a 15 minute online questionnaire and a one-on-one website task and follow-up interview. The task will take approximately 45 minutes and will occur at a time and location that is most convenient to you. Upon completion of the interview, participants will receive a gift certificate to Chapters.
Practicing Teachers Needed for Research Study on Educational Websites
177
Appendix A-5
Demographic Characteristics across the Three Different Modes of Recruitment
Demographic
Characteristic
Recruitment Type
Poster AQ Course Word of Mouth
Gender
Male 6 1 2
Female 22 3 11
Age
25-29 5 2 2
30-34 10 0 2
35-39 8 1 2
40-44 1 1 4
45-49 2 0 1
50-54 1 0 1
55+ 1 0 1
Years Teaching
2-4 years 6 2 2
5-9 years 14 0 3
10-14 years 6 2 4
15-19 years 1 0 1
20-24 years 1 0 1
25+ years 0 0 2
Current Grade
Kindergarten 1 0 1
Primary 14 0 3
Junior 9 1 9
Primary/Junior 4 3 0
Comfort Using
Internet
Very
Comfortable
25 3 13
Somewhat
Comfortable
3 1 0
Not Very
Comfortable
0 0 0
Frequency of
Internet Use for
Professional
Purposes
> once/day 9 1 4
Once/day 13 1 7
Once/week 5 2 2
Once/month 1 0 0
178
Appendix A-6
Demographic Questionnaire
1. Which grade do you currently teach?
□ Kindergarten □ Grade 4
□ Grade 1 □ Grade 5
□ Grade 2 □ Grade 6
□ Grade 3
2. What grade(s) have you taught in your teaching career?
□ Kindergarten □ Grade 4
□ Grade 1 □ Grade 5
□ Grade 2 □ Grade 6
□ Grade 3
3. What is your gender?
Male
Female
4. What is your age-range?
□ 20-25 years □ 41-45 years
□ 26-30 years □ 46-50 years
□ 31-35 years □ 51-55 years
□ 36-40 years □ 56 + years
5. How long have you been teaching at the elementary level?
years
6. What technology devices do you own?
Desktop Computer
Laptop Computer
PDA (personal data assistant)
179
Ipod/MP3 Player with Internet Access
Smartphone: _____________
Tablet PC
iPad
eReader Device
Other: __________________________________________________________
7. What technology devices do you use?
Desktop Computer
Laptop Computer
PDA (personal data assistant)
Ipod/MP3 Player with Internet Access
Smartphone: _____________
Tablet PC
iPad
eReader Device
Other: __________________________________________________________
8. Do you use the Internet for your teaching practices?
□ Yes
□ No
(a) If yes, please indicate how you use the Internet.
9. How comfortable do you feel using the Internet?
□ Very comfortable
□ Somewhat comfortable
□ Not very comfortable
□ Not comfortable at all
180
10. How frequently do you use the Internet for your teaching practices?
Once a month
Once a week
Once a day
More than once a day
11. What percent of the time would you say you spend using the Internet to plan your literacy
lessons/program?
________%
12. What resources do you use when planning for your classroom set-up/literacy lessons?
□ Books
□ Internet
□ Literacy Coach
□ Workshop Material
□ Professional Learning Community Material
Other:
13. What three websites do you use most often in your teaching practices?
(i)
(ii)
(iii)
14. Why do you like these websites?
15. Which of the following website tools do you use when exploring educational websites?
□ Video clips
□ Photographs
181
□ Blogs
□ Chat
□ Discussion boards
□ Other:
16. Which website tools do you find most beneficial to your teaching practices or placements?
□ Video clips
□ Photographs
□ Blogs
□ Chat
□ Discussion boards
□ Other:
17. How passionate are you about teaching literacy? Please circle one.
1 2 3 4 5 6 7
Not at all Extremely
18. How confident do you feel about teaching literacy? Please circle one.
1 2 3 4 5 6 7
Not at all Extremely
19. How confident do you feel about using website features/tools? Please circle one.
1 2 3 4 5 6 7
Not at all Extremely
20. How often do you attend workshops or conferences? Please circle one.
1 2 3 4
Never Not often Often Very often
(a) What are your reasons for your choice you made in question 23?
21. How often do you attend workshops for literacy? Please circle one.
1 2 3 4
Never Not often Often Very often
182
22. How motivated are you to incorporate a new teaching strategy that you have learned about
into your future program?
1 2 3 4 5 6 7
Not at all Extremely
Thank you
183
Appendix A-7
Think Aloud and Interview Consent Form
Thank you for participating in this research study. During this session, you will be asked to
navigate through a professional development website and think aloud about your navigational
experience. A list of questions will then be asked concerning your experience of exploring the
website as well as your general feelings towards thinking aloud. You may skip any questions if
you do not want to answer them. You may choose to end the interview at any time, without
consequences. The think aloud and interview will be audio-recorded so that we can later
transcribe it to ensure we do not lose any valuable information.
Your signature below indicates that you understand that the think aloud and interview will be
audio-recorded and that you give your consent for this.
_____________________________________ _________________________________
Name (please print) Signature
_ ______
Date
184
Appendix A-8
Screen Shot of Camtasia Studio During a Participant’s Navigation
185
Appendix A-9
Interview Questions
1. What were your general feelings while you explored the Balanced Literacy Diet website?
2. On a scale of 1 to 7, 1 indicating very uncomfortable and 7 indicating very comfortable, how
did you feel about thinking aloud while you were navigating/following your
navigation/while you watched a screen recording of your navigation? Why did you give
yourself this rating?
3. What did you find challenging while you explored the Balanced Literacy Diet website?
4. What features of the Balanced Literacy Diet website did you find most appealing?
5. What other features would you have liked to view/explore in the Balanced Literacy Diet
website?
6. Is there anything else you would like to share about online professional resources?
7. Is there anything else you would like to share about your experience of thinking aloud while
exploring The Balanced Literacy Diet website?
186
Appendix B-1
Initial List of Potential Categories for Coding Think Aloud Transcripts
Potential Categories
1. Impression category
2. Information content (textual, multimedia)
3. Layout (text, graphic elements)
4. Information architecture (structural design)
a. Links
b. Navigation
c. Structure
5. Diversion
a. Confusion
b. Loading time
6. Reading-reading words, phrases or sentences off the screen; reading texts, headings and
links
7. Procedure-describing current or future activity
8. Observation-making an observation about the website or participant’s own behaviour
9. Explanation-explaining, using either present or future tense, the reason for participant’s
behaviour; providing insight into the participant’s motivation for his/her behaviour
10. Verbal fillers
11. Identifying and making sense of new information
12. Evaluating claims and evidence
13. Considering ways to use the site in their own practice
14. Intentions-information representing goals-future states of the subject-shall, will, must
15. Cognitions-information based on attention to selected aspects of the current situation
16. Planning-information representing intermediate constructions to explore sequences of
possibilities
17. Evaluation-explicit or implicit comparisons
18. Comprehension
19. Problem solving
20. Concept formation
21. Prior knowledge
22. Decision making
23. An evaluation of an aspect
24. A causal relation
25. Collection of information
26. Evaluation of information
27. Strategy
28. Choice
29. Relative evaluation of options
30. Attention-refers to the processing of information on the site
31. Perception: relates to the perception of text and graphics on the site
32. Problem-solving: refers to the provision of support for using the sites
33. Memory: in relation to the retrieval of information on the site
34. Cognitive mapping: taking in sensory information (e.g., I see…)
a. Sensory information comprised of:
187
i. Spatial characteristics
ii. Signing of website (hyperlinks, navigational bar)
b. Inferential information:
i. Comments on website environment
c. Memory information:
i. Seldom recalled, when did: “other commercial sites”
35. Decision generation: subjects would often pause before making a decision
36. Action description-describe what they were doing or going to do or just did
37. Action explanation-explain the reason(s) for executing certain actions
38. Result evaluation-summarize understanding or give evaluation of content, links or the
outcome of an action
39. User experience-express positive or negative feelings, aesthetic preferences towards the
websites
40. Problem formulation-verbalize difficulties, including utterances that participants indicate
uncertainty; and utterances that participants not only express a negative feeling or
disapproval, but also indicate that it was caused by system based issue(s)
41. Causal explanation-explain what caused certain outcome or event
42. Recommendation-to improve site
43. Procedural behaviour- specific statements about what subjects did during their task
performance, such as “I read the instructions at the top”
44. Inferential/explanatory statements
a. Logic inference-directly inferred from or generated based on information that users
attended to, such as “I see the top and bottom of this”
b. Perception explanation-such as, “the picture is pretty bright”
c. Strategy explanation—about how subjects completed the task, such as “and this one I
just started doing the subtraction”
45. Reflective comments-contain self-diagnostics about what subjects did or should have done,
such as “for this one I was actually a little confused about what they were asking at first”
46. Meta-comments-such as, “this one kind of took me by surprise”
47. Evaluative statements: “are related to cognitive processes that link pieces of information
obtained through experience and stored in memory of the decision maker”
48. Predictive statements: reference to a future action
49. Decisions: simple statement of the selection of an option
50. Understand-construct meaning from instructional messages, including written and graphic
a. Interpreting-clarifying, paraphrasing, representing
b. Exemplifying-illustrating (finding a specific example or a concept
c. Classifying-categorizing
d. Summarizing-abstracting, generalizing
e. Inferring-concluding, extrapolating (drawing a logical conclusion from presented
information
f. Comparing-contrasting, mapping, matching
g. Explaining-constructing models
51. Analyze: break material into its constituent parts and determine how the parts relate to one
another and to an overall structure or purpose
a. Differentiating-discriminating, distinguishing (relevant from irrelevant parts,
important form unimportant parts of presented material), focusing, selecting
b. Organizing-finding coherence, integrating, outline, structuring (determining how
elements fit or function within a structure)
188
52. Evaluate-make judgements based on criteria and standards
a. Checking-coordinating, monitoring
b. Critiquing-judging
53. Create-put elements together to form a coherent whole; reorganize elements into a new
patter or structure
a. Generating-hypothesizing
b. Planning-designing
189
Appendix B-2
Krathwohl’s Revised Taxonomy
Structure of the Cognitive Process
Dimension of the Revised Taxonomy
(based on Krathwohl, 2002)
1.0 Remember—Retrieving relevant knowledge from long-term memory.
1.1 Recognizing
1.2 Recalling
2.0 Understand—Determining the meaning of instructional messages, including oral, written,
and graphic communication.
2.1 Interpreting
2.2 Exemplifying
2.3 Classifying
2.4 Summarizing
2.5 Inferring
2.6 Comparing
2.7 Explaining 3.0 Apply—Carrying out or using a procedure in a given situation.
3.1 Executing
3.2 Implementing 4.0 Analyze—Breaking material into its constituent parts and detecting how the parts relate to
one another and to an overall structure or purpose.
4.1 Differentiating
4.2 Organizing
4.3 Attributing 5.0 Evaluate—Making judgements based on criteria and standards.
5.1 Checking
5.2 Critiquing 6.0 Create—Putting elements together to form a novel, coherent whole or make an original
product.
6.1 Generating
6.2 Planning
6.3 Producing
190
Appendix B-3
Frequency Counts of Web-based Actions
Concurrent Condition
Behaviour/Action C0
1
C0
2
C0
3
C0
4
C0
5
C0
6
C0
7
C0
8
C0
9
C1
0
C1
1
C1
2
C1
3
C1
4
C1
5
Enters a search term 7 3 0 1 0 0 0 0 0 0 0 0 1 3 2
Maneuvers around virtual
tour 0 0 0 2 5 0 0 3 0 0 4 0 0 0 0
Opens content page 3 1 0 4 0 0 5 0 2 0 4 1 4 0 0
Opens food groups page 8 5 8 6 2 2 4 2 6 5 7 2 12 11 16
Opens homepage 4 1 1 2 2 0 2 1 2 4 3 0 6 2 2
Opens how-to videos 0 2 0 1 1 2 1 2 0 2 1 0 1 2 0
Opens outside link 11 1 1 4 0 4 1 1 2 0 2 0 5 0 0
Opens recipe 10 13 8 8 2 7 10 4 5 3 6 8 2 7 9
Opens recipe finder page 7 4 5 8 2 9 3 4 1 2 2 7 2 8 0
Opens virtual tour 3 4 1 4 2 1 1 1 2 1 1 0 4 0 0
Selects filter option 4 5 2 0 0 0 1 0 1 0 3 0 0 2 0
Starts a video 6 4 2 7 8 3 5 7 2 8 4 2 4 4 5
Stops video before the
end 4 4 2 6 7 2 5 8 2 8 4 1 4 4 4
Takes a note 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Uses virtual tour arrow 4 6 1 11 2 0 0 0 3 0 0 0 3 0 0
Uses virtual tour zoom 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
Views a photograph 0 8 9 0 0 6 4 8 0 6 9 7 0 20 9
Views a video 13 5 5 22 40 9 4 38 11 19 10 9 10 8 32
Total Number of Pages 46 32 25 37 12 26 30 16 22 18 26 18 37 31 28
191
Retrospective Condition
Behaviour/Action R01 R02 R03 R04 R05 R06 R07 R08 R09 R10 R11 R12 R13 R14 R15
Enters a search term 0 0 0 0 3 1 0 0 1 0 0 0 1 0 0
Opens content page 1 1 1 1 1 2 1 0 1 0 1 0 3 0 0
Opens food groups page 3 1 2 5 3 3 2 3 1 2 4 2 5 3 3
Opens homepage 0 0 0 1 1 1 0 0 1 1 1 0 1 1 1
Opens how-to video 0 0 0 0 1 1 0 0 1 1 0 1 1 0 1
Opens outside link 4 1 0 0 0 1 0 0 1 0 0 0 1 2 2
Opens recipe 3 9 17 6 10 6 10 5 4 6 4 5 2 3 2
Opens recipe finder page 3 6 0 2 7 2 3 0 1 3 3 2 1 0 15
Opens virtual tour 1 0 1 1 1 1 1 1 2 4 2 1 1 1 0
Selects filter option 0 1 0 0 0 0 0 0 7 0 0 0 0 0 5
Starts video 3 0 2 7 3 3 2 6 2 8 1 6 4 4 2
Stops video before the end 1 0 1 3 3 0 2 3 1 3 0 0 0 0 1
Takes a note 0 0 0 1 0 0 0 8 0 0 0 0 0 0 0
Total number of pages 15 18 22 16 23 17 17 10 12 17 15 11 15 11 23
Uses virtual tour arrows 0 0 0 0 2 0 6 0 5 6 0 0 4 0 0
Uses virtual tour
maneuver 9 0 0 0 0 0 2 0 0 0 0 0 1 0 0
Uses virtual tour zoom 3 0 0 0 0 0 1 2 1 0 0 0 0 1 0
Views a photograph 15 9 37 2 17 16 7 0 0 11 0 0 1 0 6
Views video 32 0 12 65 23 30 19 47 12 56 8 66 64 46 11
192
Virtual Revisit Condition
Behaviour/Action V01 V02 V03 V04 V05 V06 V07 V08 V09 V10 V11 V12 V13 V14 V15
Enters a search term 0 0 0 0 0 0 0 1 0 0 2 2 1 0 0
Opens content page 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0
Opens food groups page 2 5 3 4 0 4 5 1 4 3 2 1 2 7 1
Opens home page 3 1 0 0 1 1 1 0 2 0 0 0 0 0 1
Opens how-to video 0 0 0 1 1 0 0 1 0 0 1 0 2 2 0
Opens outside link 0 4 0 1 0 3 1 0 0 0 1 5 1 0 1
Opens recipe 5 10 0 4 9 10 6 6 6 2 8 8 6 5 9
Opens recipe finder page 5 8 1 4 9 8 9 5 0 2 5 4 8 4 6
Opens virtual tour 2 2 1 1 1 0 2 1 2 2 1 2 1 1 1
Selects filter option 0 2 0 1 0 0 6 2 0 0 1 1 3 1 0
Starts video 1 3 7 6 4 4 2 9 4 7 2 2 7 15 8
Stops video before the end 0 2 0 4 0 3 2 4 3 2 1 0 3 12 2
Takes a note 0 0 8 0 0 0 10 0 0 0 3 0 0 0 2
Total Number of Pages 18 31 7 16 23 25 25 15 16 11 19 21 21 21 19
Uses virtual tour arrows 12 2 0 8 0 0 0 1 1 0 0 10 2 0 0
Uses Virtual tour
maneuver 0 0 0 0 0 0 0 0 2 3 5 0 0 0 0
Uses virtual tour zoom 0 0 0 0 0 0 0 0 2 0 0 1 0 0 0
Views a photograph 17 3 0 8 10 24 0 0 24 0 7 0 0 0 0
Views video 9 35 69 47 29 35 6 61 26 45 8 14 43 48 59
193
Appendix B-4
Sample Observations of Web-based Actions with Think Aloud Recordings
Concurrent Condition
*Note: SV = Silent interval, Video playing
0:10 0:20 0:30 0:40 0:50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 2:40
C01 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C02 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C03 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C04 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C05 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C06 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C07 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C08 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y SV
C09 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C10 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C11 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C12 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C13 Y Y Y Y Y Y Y Y Y Y Y Y SV Y Y Y
C14 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
C15 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
Retrospective Condition:
0:10 0:20 0:30 0:40 0:50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 2:40
R01 Y Y Y N N N N Y N N N N N N N N
R02 N N N N N N Y Y Y Y Y Y Y Y Y Y
R03 N N N Y N N N N N N N N N N N N
R04 N N N N N N N N N N Y Y Y Y N N
R05 Y Y Y N N N N N N N N N N N N N
R06 N N N Y Y Y N N N N N N N N N N
R07 Y Y Y Y Y N N N N N N N N N N N
R08 N N N N N N N N N N N N N N N N
R09 Y Y N N N N N N N N N N N N N N
R10 Y Y Y Y Y Y Y N N N N N N N N N
R11 N N N N N N N N N N N
R12 N N N N N N N N N N N N N N N N
R13 Y Y Y Y Y Y Y Y N N N
R14 Y Y Y Y Y Y N N N N N N N N N N
R15 Y Y Y Y Y N N N N N N N N N N
194
Virtual Revisit Condition:
0:10 0:20 0:30 0:40 0:50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 2:40
V01 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V02 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V03 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V04 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V05 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V06 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V07 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V08 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V09 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V10 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V11 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V12 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V13 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V14 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
V15 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
195
Appendix B-5
Data Screening Results and Descriptives
N Mean Skewness Kurtosis 95% Confidence
Interval for Mean
Group Stat. SE Stat. SE Stat. SE Lower
Bound
Upper
Bound
Planning 1 15 0.41 0.18 1.75 0.58 2.25 1.12 0.02 0.80
2 15 3.96 0.99 0.48 0.58 -1.20 1.12 1.94 6.08
3 15 4.17 0.94 1.18 0.58 -0.03 1.12 2.15 6.20
Connecting 1 15 6.04 0.93 -0.04 0.58 -1.12 1.12 4.05 8.03
2 15 13.23 1.69 0.22 0.58 -0.70 1.12 9.61 16.85
3 15 9.08 0.98 0.10 0.58 -1.20 1.12 6.97 11.19
Reasoning 1 15 2.23 0.64 1.22 0.58 0.22 1.12 0.87 2.60
2 15 10.99 1.97 0.58 0.58 1.20 1.12 6.75 15.22
3 15 19.27 2.36 -0.06 0.58 -0.85 1.12 14.20 24.35
Reflecting 1 15 16.71 1.99 0.60 0.58 0.04 1.12 12.45 20.98
2 15 26.86 2.96 -0.15 0.58 -0.54 1.12 20.52 33.20
3 15 25.08 1.93 0.99 0.58 0.83 1.12 20.94 29.22
Website
Content
1
15
10.87
1.28
0.56
0.58
-0.25
1.12
8.12
13.61
2 15 12.18 2.25 -0.82 0.58 -1.26 1.12 7.36 17.00
3 15 15.36 1.36 -0.21 0.58 -0.88 1.12 12.44 18.28
User
Experience
1
15
7.96
1.80
1.07
0.58
0.82
1.12
4.09
11.83
2 15 12.13 2.33 1.20 0.58 3.20 1.12 7.14 17.13
3 15 7.47
0.85 0.16 0.58 -0.36 1.12 5.64 9.30
Understanding 1 15 5.99 0.90 0.73 0.58 -0.24 1.12 4.05 7.93
2 15 2.97 1.15 1.95 0.58 4.23 1.12 0.50 5.43
3 15 3.36 0.81 1.47 0.58 2.13 1.12 1.62 5.10
Diversion 1 15 5.97 0.78 0.04 0.58 -1.88 1.12 4.30 7.64
2 15 0.79 0.54 2.41 0.58 4.25 1.12 0.36 1.94
3 15 2.99 0.77 1.55 0.58 2.27 1.12 1.34 4.64
Procedural
Behaviour
1
15
11.64
1.60
0.56
0.58
0.05
1.12
8.20
15.08
2 15 11.71 2.35 0.52 0.58 0.01 1.12 6.66 16.76
3 15 9.37 1.27 0.14 0.58 -0.53 1.12 6.65 12.09
Website
Features
1
15
10.15
1.26
0.24
0.58
-1.13
1.12
7.45
12.84
2 15 3.49 1.25 1.24 .058 1.12 1.12 0.82 6.17
3 15 3.56
0.91 1.98 0.58 3.81 1.12 1.60 5.52
Reading 1 15 22.03 3.30 0.27 0.58 -0.38 1.12 14.95 29.12
2 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
3 15 0.29 0.16 1.97 0.58 2.50 1.12 0.06 0.65
Word Count
1
15
1676.13
101.96
0.42
0.58
0.93
1.12
1457.45
1894.82
2 15 658.80 59.13 0.23 0.58 -0.38 1.12 531.98 785.62
3 15 2637.87 92.94 -0.98 0.58 0.56 1.12 2438.54 2837.20
196
Number of
Thought Units
1 15 147.40 9.25 0.56 0.58 0.77 1.12 127.56 167.24
2 15 30.73 3.40 0.60 0.58 -0.14 1.12 23.45 38.02
3 15 123.27 5.31 0.90 0.58 0.64 1.12 111.87 134.66
Time on Each
Page
1
15
36.07
2.35
0.32
0.58
0.02
1.12
31.04
41.10
2 15 52.53 3.90 0.08 0.58 -0.84 1.12 44.17 60.90
3 15 47.80 3.07 0.42 0.58 -0.99 1.12 41.23 54.37
Number of
Pages Visited
1
15
26.93
2.33
0.36
0.58
0.03
1.12
21.94
31.93
2 15 16.13 1.08 0.32 0.58 -0.65 1.12 13.81 18.45
3 15 18.60 1.37 -0.40 0.58 0.77 1.12 15.66 21.54
Food Group
Pages
1
15
6.40
1.06
0.95
0.58
0.61
1.12
4.12
8.68
2 15 2.80 0.31 0.44 0.58 -0.07 1.12 2.13 3.47
3 15 2.93 0.49 0.47 0.58 -0.18 1.12 1.88 3.99
Content Pages 1 15 1.60 0.49 0.95 0.58 -1.29 1.12 0.56 2.64
2 15 0.87 0.22 1.13 0.58 1.96 1.12 0.40 1.33
3 15 0.40 0.13 0.46 0.58 -2.09 1.12 0.12 0.68
Visits to
Homepage
1
15
2.13
0.41
0.96
0.58
1.20
1.12
1.25
3.02
2 15 0.60 0.13 -0.46 0.58 -2.09 1.12 0.32 0.88
3 15 0.67 0.23 1.46 0.58 2.01 1.12 0.17 1.16
Time Viewing
Videos
1
15
150.67
32.30
1.08
0.58
-0.05
1.12
81.39
219.94
2 15 334.00 60.68 0.21 0.58 -1.61 1.12 203.86 464.14
3 15 355.33 51.72 -0.83 0.58 -1.08 1.12 244.40 466.26
Number of
Videos
Stopped
1
15
4.27
0.53
0.27
0.58
-0.62
1.12
3.13
5.40
2 15 1.20 0.33 0.55 0.58 -1.42 1.12 0.50 1.90
3 15 2.53 0.77 2.47 0.58 7.78 1.12 0.89 4.18
Enters Search
Term
1
15
1.13
0.51
2.22
.058
5.36
1.12
0.05
2.22
2 15 0.40 0.21 2.54 0.58 7.07 1.12 0.06 0.86
3 15 0.40 0.19 1.63 0.58 1.32 1.12 0.01 0.81
Maneuvers
Virtual Tour
1
15
0.93
0.44
1.60
0.58 1.24 1.12 0.01 1.88
2 15 0.80 0.60 3.54 0.58 12.94 1.12 0.49 2.09
3 15 0.67 0.39 2.29 0.58 4.74 1.12 0.16 1.50
Visits How-To
Videos
1
15
1.00
0.22
.00
0.58
-1.62
-1.20
0.63
1.37
2 15 0.47 0.13 0.15 0.58 -2.31 1.12 0.09 0.84
3 15 0.53 0.19 1.07 0.58 -0.11 2.20 0.16 0.91
Visits Outside
Link
1
15
2.13
0.76
2.18 0.58 5.51 1.12 0.50 3.77
2 15 0.80 0.30 1.76 0.58 3.37 1.12 0.17 1.43
3 15 1.13 0.41 1.57 0.58 1.54 1.12 0.25 2.02
Views Recipe
Page
1
15
6.80
0.82
0.04
0.58
-0.43
1.12
5.05
8.55
2 15 6.13 1.03 1.57 0.58 2.93 1.12 3.93 8.34
3 15 6.27 0.74 -0.65 0.58 0.18 1.12 4.68 7.85
Views Recipe
Finder Page
1
15
4.28
0.75
0.31 0.58 -1.32 1.12 2.67 5.87
2 15 3.20 0.99 2.32 0.58 6.36 1.12 1.07 5.33
197
3 15 5.20 0.73 -0.28 0.58 -0.78 1.12 3.63 6.77
Visits Virtual
Tour
1
15
1.67
0.37
0.68
0.58
-0.85
1.12
0.87
2.47
2 15 1.20 0.24 1.92 0.58 5.54 1.12 0.68 1.72
3 15 1.33 0.16 -0.31 0.58 -0.40 1.12 0.99 1.68
Selects Filter
Option
1
15
1.20
0.43
1.26 0.58 0.58 1.12 0.28 2.12
2 15 0.87 0.55 2.49 0.58 5.32 1.12 0.31 2.05
3 15 1.13 0.42 2.10 0.58 5.09 1.12 0.22 2.04
Takes a Note 1 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
2 15 0.60 0.53 3.78 0.58 14.42 1.12 0.54 1.74
3 15 1.53 0.82 2.15 0.58 3.68 1.12 0.23 3.30
Virtual Tour
Arrow
1
15
2.00
0.81 2.01 0.58 4.34 1.12 0.27 3.73
2 15 1.53 0.62 1.16 0.58 -0.44 1.12 0.20 2.87
3 15 2.40 1.05 1.69 0.58 1.49 1.12 0.15 4.65
Views Photos
1
15
5.73
1.41
1.07
0.58
2.14
1.12
2.72
8.75
2 15 8.07 2.63 1.76 0.58 3.75 1.12 2.42 13.71
3 14 6.64 2.40 1.18 0.60 0.08 1.15 1.47 11.82
Comfort Level 1 15 5.07 0.32 -0.14 0.58 -0.36 1.12 4.39 5.74
2 15 6.07 0.25 -0.71 0.58 -0.33 1.12 5.53 6.60
3 15 6.27 0.23 -1.32 0.58 1.82 1.12 5.78 6.76
*Note: Group 1 = Concurrent, 2 = Retrospective, 3 = Virtual Revisit
198
Tests of Normality
Kolmogorov-Smirnov(a) Shapiro-Wilk
Group Statistic df Sig. Statistic df Sig.
Planning 1 0.39 15 .000 0.66 15 .000
2 0.18 15 .191 0.88 15 .044
3 0.30 15 .001 0.80 15 .004
Connecting 1 0.20 15 .101 0.94 15 .392
2 0.12 15 .200* 0.96 15 .744
3 0.15 15 .200* 0.95 15 .513
Reasoning 1 0.31 15 .000 0.80 15 .004
2 0.14 15 .200* 0.94 15 .328
3 0.14 15 .200* 0.96 15 .722
Reflecting 1 0.13 15 .200* 0.96 15 .628
2 0.12 15 .200* 0.98 15 .950
3 0.15 15 .200* 0.89 15 .063
Website
Content
1
0.10
15
.200*
0.95
15
.504
2 0.12 15 .200* 0.93 15 .287
3 0.13 15 .200* 0.95 15 .545
User
Experience
1
0.16
15
.200*
0.89
15
.065
2 0.17 15 .200* 0.88 15 .050
3 0.14 15 .200* 0.96 15 .670
Understanding 1 0.20 15 .119 0.92 15 .219
2 0.28 15 .002 0.73 15 .000
3 0.20 15 .097 0.86 15 .024
Diversion 1 0.24 15 .023 0.86 15 .025
2 0.51 15 .000 0.41 15 .000
3 0.23 15 .029 0.84 15 .011
Procedural
Behaviour
1
0.17
15
.200*
0.96
15
.607
2 0.13 15 .200* 0.94 15 .381
3 0.14 15 .200* 0.97 15 .875
Website
Features
1
0.15
15
.200*
.946
15
.463
2 0.30 15 .001 .768 15 .001
3 0.27 15 .004 .739 15 .001
Reading 1 0.14 15 .200* .964 15 .762
2 - - - - - -
3 0.48 15 .000 .520 15 .000
Word Count 1 0.13 15 .200* 0.95 15 0.48
2 0.17 15 .200* 0.95 15 0.50
3 0.18 15 .200* 0.93 15 0.23
Number of
Thought Units
1
0.12
15
.200*
.971
15
.878
2 0.15 15 .200* .939 15 .366
3 0.17 15 .200* .934 15 .311
Time on Each
Page
1
0.17
15
.200*
.971
15
.869
2 0.09 15 .200* .969 15 .842
3 0.19 15 .137 .935 15 .326
Number of
Pages Visited
1
0.09
15
.200*
.983
15
.987
2 0.15 15 .200* .924 15 .219
199
3 0.13 15 .200* .967 15 .818
Food Group
Pages
1
0.15
15
.200*
.903
15
.106
2 0.23 15 .026 .903 15 .107
3 0.15 15 .200* .956 15 .627
Content Pages 1 0.27 15 .005 .793 15 .003
2 0.30 15 .001 .794 15 .003
3 0.39 15 .000 .630 15 .000
Visits to
Homepage
1
0.27
15 .005 .900 15 .096
2 0.39 15 .000 .630 15 .000
3 .304 15 .001 .748 15 .001
Time Viewing
Videos
1
0.23
15
.029
.851
15
.018
2 0.17 15 .200* .896 15 .466
3 0.13 15 .200* .946 15 .466
Number of
Videos
Stopped
1
0.22
15
.053
.927
15
.246
2 0.23 15 .033 .790 15 .003
3 0.24 15 .016 .717 15 .000
Enters Search
Term
1
0.32
15 .000 .657 15 .000
2 0.42 15 .000 .557 15 .000
3 0.44 15 .000 .596 15 .000
Maneuvers
Virtual Tour
1
0.44 15 .000 .616 15 .000
2 0.43 15 .000 .403 15 .000
3 0.47 15 .000 0.53 15 .000
Visits How-
To Videos
1
0.22
15 .061 .805 15 .004
2 0.35 15 .000 .643 15 .000
3 0.36 15 .000 .716 15 .000
Visits Outside
Link
1
0.25 15 .012 .730 15 .001
2 0.29 15 .001 .734 15 .001
3 .333 15 .000 .728 15 .001
Views Recipe
Page
1
0.13
15
.200*
.962
15
.723
2 0.25 15 .015 .850 15 .018
3 0.14 15 .200* .937 15 .343
Views Recipe
Finder Page
1
0.18 15 .186 .920 15 .190
2 0.32 15 .000 .735 15 .001
3 0.17 15 .200* .934 15 .312
Visits Virtual
Tour
1
0.28
15
.003
.845
15
.015
2 0.31 15 .001 .712 15 .001
3 0.31 15 .001 .766 15 .001
Selects Filter
Option
1
0.30 15 .001 .766 15 .001
2 0.46 15 .000 .476 15 .000
3 0.27 15 .005 .723 15 .000
200
Takes a Note 1 - - - - - -
2 0.48 15 .000 .332 15 .000
3 0.42 15 .000 .561 15 .000
Virtual Tour
Arrow
1
0.27 15 .004 .711 15 .000
2 0.40 15 .000 .660 15 .000
3 0.34 15 .000 .649 15 .000
Views Photos
1
0.21
15
.081
.842
15
.013
2 0.21 15 .062 .790 15 .003
3 0.27 14 .006 .757 14 .002
Comfort
Level
1
0.21
15 .069 .922 15 .205
2 0.23 15 .026 .844 15 .014
3 0.26 15 .006 .775 15 .002
* This is a lower bound of the true significance.
(a) Lilliefors Significance Correction
*Note: Group 1 = Concurrent, 2 = Retrospective, 3 = Virtual Revisit
201
Test of Homogeneity of Variance
Levene Statistic df1 df2 Sig.
Planning 14.36 2 42 .000
Connecting 3.44 2 42 .041
Reasoning 6.04 2 42 .005
Reflecting 2.11 2 42 .134
Website Content 5.02 2 42 .011
User Experience 2.43 2 42 .101
Diversion 2.98 2 42 .062
Procedural Behaviour 2.20 2 42 .124
Website Features 1.86 2 42 .169
Reading 21.04 2 42 .000
Word Count 0.97 2 42 .389
Number of Thought Units 4.99 2 42 .011
Time on Each Page 2.59 2 42 .087
Number of Pages Visited 3.870 2 42 .029
Food Groups Pages 7.46 2 42 .002
Content Pages 19.83 2 42 .000
Visits to Homepage 3.32 2 42 .046
Time Viewing Videos 4.61 2 42 .015
Number of Videos Stopped 0.96 2 42 .393
Enters Search Term 4.38 2 42 .019
Maneuvers Virtual Tour 0.19 2 42 .826
Visits How-To Videos 1.05 2 42 .358
Visits Outside Link 3.08 2 42 .057
Views Recipe Page .382 2 42 .685
Views Recipe Finder Page .053 2 42 .948
Visits Virtual Tour 5.39 2 42 .008
Selects Filter Option 0.18 2 42 .837
Takes a Note 7.31 2 42 .002
Virtual Tour Arrow 1.01 2 42 .374
Views Photos 2.02 2 42 .146
Comfort Level 0.39 2 42 .677
202
Appendix C-1
Initial List of Categories for Study 2
Initial List of Categories with Frequency Counts
Category
Frequency
Count
Absorbing or mentally taking in information 20
Accessing information with ease 1
Attending to the spatial organization of the classroom 6
Being curious about website features or content 9
Comparing content to own teaching practice 49
Connecting content to special ed. classroom contexts 4
Connecting navigation and website content to current grade 46
Connecting navigational experience to professional learning 23
Connecting to teacher's college 1
Connecting website content to future planning and
classroom set up
39
Connecting website content to literacy knowledge and
understanding
24
Connecting website content to prior knowledge 6
Connecting website to other resources 5
Constructing new ideas for future practice-constructing
knowledge
21
Continuing navigation 1
Critiquing website content and organization 15
Deciding to explore information in-depth 2
Describing challenges in implementing literacy program 5
Describing multitasking behaviour 4
Describing website content being viewed 3
Discontinuing an action or behaviour 20
Encoding information for future retrieval 31
Explaining online behaviour 83
Explaining what and why content was read 33
Exploring information 14
Expressing awareness of study task 17
Expressing opinion about website content 71
Expressing opinion about website features and organization 34
Feeling confident during navigation 4
Feeling confused 16
Feeling frustrated about navigation 20
Feeling inspired excited motivated 13
203
Feeling more advanced or superior than website content 2
Feeling overwhelmed 7
Focusing on PD initiatives 1
Hoping for more information 24
Identifying goals of instructional practice or best practices
in literacy
71
Identifying new knowledge new information 19
Identifying or recognizing familiar information 27
Identifying or referring to own learning style 35
Identifying professional goals 41
Identifying self as a learner 9
Learning by observing, watching 5
Looking for representation 2
Looking for resources for specific literacy concept 1
Maintaining current knowledge and understanding of
literacy practices
4
Making a navigational decision 52
Making a navigational decision based on own children's
struggles
2
Narrowing the focus down 6
Navigating with ease 2
Noticing appealing and eye catching features 24
Noticing new information 2
Noticing website errors 1
Observing and reflecting on own actions and non-actions 31
Offering improvements to website content 6
Planning navigation-thinking ahead about options,
navigational strategies
7
Reading to find new or interesting information 2
Realizing or discovering how to use certain website features 9
Referring to future use of website 6
Referring to limitations in own professional development
OR how technology can be used to off set PD limitations
5
Referring to literacy terms and or practices 4
Reflecting on general use of websites-online learning style 33
Reflecting on program 3
Reflecting on students' needs and goals 13
Relating content to a stage or level of teaching 27
Relating content to real-life or authentic experiences of
classroom practice
12
Relating or associating website content and pages to each
other
4
204
Revisiting information to refresh professional knowledge 15
Sharing a learning experience of visiting colleagues'
classrooms
5
Tailoring information to teaching practice 10
Troubleshooting strategies 6
Trying to understand and use website features and
organization
31
Using background information and prior knowledge to
understand content
3
Wanting to understand the credibility of the website 8
Willing to move outside comfort zone 1
Wondering about activity extensions 2
Wondering about classroom contexts 4
205
Appendix C-2
Sample of Memos Written During Analysis
06/15/2014
Initial Coding:
Coding first five transcripts (V01-V05) without looking back and regrouping or reorganizing
codes. All codes are being laid out on the table and new codes are being added as they emerge
from the transcripts. Using verbs to begin codes (comparing, connecting, constructing)-what
Charmaz refers to as "activating coding"-coding with words that reflect action
06/30/2014
Categories:
Categories may include broad areas, information directly relating to website content and
organization and information relating to cognition and learning
07/05/2014
Teacher Knowledge and Understanding of Instructional Practice:
Learning about teachers' understandings of what they view as best practices in literacy. They are
often expressing research based practices in relation to what they see on the website and their
current grade. Also relating these best practices to their professional goals and school wide
goals.
08/23/2014
Themes/Model:
Analysis of think aloud transcripts + interviews + questionnaire data = understanding of
elementary teachers' motivations and beliefs in literacy planning and practice
09/20/2014
Triangulation:
Use initial codes from think aloud analysis to code/analyze interviews and questionnaire
(triangulation)-while coding keep an open mind about new categories
09/29/2014
Use of Internet for Professional Practice:
From Questionnaire: All teachers in VR group reported that they use the Internet for their
professional practice--> motivation to seek out new and relevant material and also to incorporate
technology into their daily practice, with their students.
206
Appendix C-3
Themes and Descriptions Sent to Participants Involved in Member Check
Potential Themes
THEMES IDENTIFIED:
1. Focusing on student needs and goals
Regardless of experience and background, teachers approached the website with the
common goal of supporting their students
2. Analyzing and evaluating information in relation to their own practice
Connecting, analyzing Analysing and deconstructing content in relation to experience and
background
Reasoning-most common reason for making a decision-connecting to practice Approaching
information from various backgrounds and perspectives (spec. ed., music, drama, grade
levels)
3. Encoding information for future retrieval
Use of memory aids and strategies for future retrieval (note taking, connecting to practice)
4. Intentions for practice and professional learning goals
Intentions for practice planning, connecting
Professional development, learning goals, professional knowledge
5. Reflecting on their learning and use of website
Identifying learning styles (e.g., visual learner)
Resolving confusion with self-questioning, troubleshooting strategies
Monitoring own cognitive processes during task
Individual learning styles (individual differences in web seeking behaviours)
6. Valuing best practices in literacy and teaching
Attitudes towards teaching and learning, knowledge about literacy instruction
Beliefs about literacy instruction and overall best teaching practices
7. Feeling motivated for learning and applying new knowledge Intrinsic motivation-being curious, interested, intrigued, motivated, driven by interest
Self-motivated, self-determined
207
Appendix C-4
Participants’ ‘Member Check’ Comments
Potential Themes
THEMES IDENTIFIED:
Focusing on student needs and goals
Regardless of experience and background, teachers approached the website with the
common goal of supporting their students
Agreed! I think this relates particularly to student engagement and also having a classroom
set-up that encourages the types of learning and interactions that we would like to
encourage.
Agree. I want to create the best program possible for my students. Yes, thinking, “what does this website have to offer me and my students? Is there anything
here that I don’t already do that I could/should be doing to help support my students in their
learning and development as readers and writers?”
Yes
Focusing on student needs AND specific curriculum expectations
I would definitely agree with this – when I search, the most important thing I would initially
be thinking about is, is this relevant and applicable to my students (age/grade/level) and will
they benefit from it?
This makes sense to me as teacher’s first goal is to support their students. It’s the first check
of how I evaluate any and all new resources.
I would hope and assume that this is a given for teachers. Do we do it without even thinking?
Analyzing and evaluating information in relation to their own practice
Connecting, analyzing yes
Analyzing and deconstructing content in relation to experience and background I especially
like seeing how teachers in the videos online introduce a topic (what language they use, how
they help the students make personal connections and engage them in the lessons)
Reasoning-most common reason for making a decision-connecting to practice I am not quite
sure that I understand this
Approaching information from various backgrounds and perspectives (spec. ed., music,
drama, grade levels)
Yes! I noticed in my transcript that I mentioned many times how I found things that I already
do. Of course I was looking for new ideas, but I think that coming across things that I
already know/do makes me think that the web site (or book or whatever) is a good fit for me
and for my teaching philosophy.
Agree. I definitely thought about how I could use some of the ideas from the site.
And deciding whether I agree or disagree with an approach. Deciding whether I think a
particular approach would work for me with my current group of students.
Yes – I was very much looking to connect to my grade level.
YES!
208
Agree – I’m always looking for things to enhance my own practice as well – so looking for
ways and ideas that might be ‘better than’ what I am already doing
Again, this makes sense to me as something must be practical and applicable to the class
you are teaching. You are first and foremost responsible for your students. Your students
and their particular understanding and needs are what drives you to explore new resources.
Content that makes sense in terms of your current practice is the easiest to utilize. It’s sort
of like ZPD. You’re most likely to move your practice to the next logical step - not jump to a
huge new overall of practice from one new resource.
Information that can be used across the curriculum and integrated into more than one
area/theme.
Encoding information for future retrieval
Use of memory aids and strategies for future retrieval (note taking, connecting to practice)
Yes. I wonder if it would be a good idea for the site to have something like a “Favourites”?
Something that lets you save or bookmark things you’d like to return to later?
Agree. I took notes.
And thinking if I have the professional knowledge, experience, time, skills and/or resources
to utilize a particular approach or to structure lessons in a new way.
Yes – I was copying links to a word document
yes – specifically ways to display information so that it is student friendly (on charts or
graphic organizers, bulletin board, in their notebooks)
Lol. This sounds like a great idea. I’m terrible at remembering where I’ve written notes.
If something really catches my eye or attention on a website, that will help me remember it
and go back to it.
Intentions for practice and professional learning goals
Intentions for practice planning, connecting organizing the ideas or concepts in a logical
order
Professional development, learning goals, professional knowledge
Always feels good to be adding to these things!
Agree. I intend to use this site to support my professional practice.
Yes, I think about future professional development, my annual learning plan/goals for
growth.
Yes
Similar to above – I agree and feel that part of my motivation is always looking for ways to
improve upon my practice and see what ideas and aids I can gather that might be better
than what I am currently doing
I think that second to student need your own professional learning goals connect to how and
when you’ll approach new resources. You have to have the impetus to want to change your
practice before you are in a position to learn / grow as an educator. Professional knowledge
connects to ZPD.
Pushing myself out of my comfort zone. Trying a new idea/method I’m not so familiar with.
Visualize myself as my students being asked to try something new/scary. That’s how growth
occurs.
209
Reflecting on their learning and use of website
Identifying learning styles (e.g., visual learner) yes
Resolving confusion with self-questioning, troubleshooting strategies yes – videos especially
show possible questions or confusions students might have with a topic
Monitoring own cognitive processes during task yes – helpful with breaking down what to
me is a simple concept but still has to be explained to students (ie: sharing what I am
thinking at various points either while reading a book or doing an experiment)
Individual learning styles (individual differences in web seeking behaviours)
Not totally sure about this one. Is this just in the sense of metacognition while using the web
site? Like thinking about how it works and how I’m using it?
Agree. This was largely a factor of the self-talk involved in the study.
Noticing my frustration or sense of calm when a sight is easy to navigate. When I click on
something and it opens instantly and I can begin to read/scan right away and not waste time
waiting. I am thinking about how the information appears visually, what I notice first, if
there are visual distractions, if it is easy to read, etc.
Yes
I think that the recorded think aloud really reinforced this.
Site was successful if return to it. Must be easy to troubleshoot.
Valuing best practices in literacy and teaching
Attitudes towards teaching and learning, knowledge about literacy instruction – especially
key literacy concepts and styles and how to teach them
Beliefs about literacy instruction and overall best teaching practices yes
Again, I think we gravitate toward things that support our existing beliefs, but it’s also great
to have something of a challenge of a piece of new learning embedded, too, so that we’re
always growing.
Agree. I told a number of colleagues about the website.
This is a big one. I spend very little time reading about instructional strategies or
approaches that I already know about. I like to quickly confirm that, yes, I am doing that in
my classroom and then read on to find out how I can do it even better or go further with it.
To me, this is similar to themes 2 and 4 again in the sense that I am motivated to improve
upon my practice and am seeking information that might help me to do so
I think your own beliefs colour everything that you do. Most people work from a positive
place of wanting the best for their students. People evaluate their understanding in terms of
what they think is best for their students, fits best with their own understanding.
Integrated into everything every day. Providing high quality materials to students.
Feeling motivated for learning and applying new knowledge
Intrinsic motivation-being curious, interested, intrigued, motivated, driven by interest also
helpful to be introduced to new material/books to teach expectations or concepts
Self-motivated, self-determined yes
Yes. I was definitely interested in exploring this site and curious about learning more!
Agree. I was excited to learn about a useful site like this.
210
Yes – I agree, curiosity, intrigue, and constantly wanting to keep my program fresh and
learn new strategies is a huge motivator.
Definitely!
Interest would certainly drive me to explore a new resource. The beauty of this website is
content is categorized / organized in so many different ways that you can explore on your
own terms.
If I’m not interested, will my students be?
General comments:
I also compare (in all honesty) my thinking around how best to teach a particular concept
or, more broadly, plan for a unit of instruction to how others are doing it. I think about how I would have categorized the information or grouped the posts to be more
in line with my sense of organization or long range plans. I am looking for posts that relate directly to overall curriculum expectations that I am
planning to teach. I am looking for ideas about how to integrate literacy across the content areas. I am looking for shortcuts so that I can work smarter, not harder. I am looking for current best practice and innovative ways of teaching concepts.
211
Appendix C-5
Frequency of Themes across Participants
Frequency Distribution of Themes Following Coding of Think Aloud Transcripts
Participant
Perception of
Professional
Learning
Focusing on
Students
Needs and
Instructional
Goals
Individual
Differences
Navigating a
Professional
Development
Website
Evaluating
Information
Encoding
Information
for Future
Retrieval
Continued
Professional
Learning
Intentions
for
Practice
Reflecting
on
Learning
V01 11 7 1 13 12 3 6 3 16
V02 3 6 8 2 10 12 1 18 10
V03 12 13 4 4 14 9 5 3 19
V04 23 4 4 7 8 1 5 7 17
V05 11 8
11 5 1 2 5 13
V06 26 4 1 2 12 1 9 1 6
V07 21 5 1 12 7 11 7 14 17
V08 20 5 4 15 12 1 5 4 26
V09 23 8 2 7 16 4 9 15 11
V10 11 5 3 11 18 5 6 2 13
V11 18 9 4 4 9 4 10 16 10
V12 17 13 4 11 12 9 11 5 23
V13 18 11 7 7 22 4 5 5 14
V14 8 9 9 8 11 1 8 5 13
V15 15 16 5 3 19 9 9 8 13
Total 237 123 57 117 187 75 98 111 221
212
Frequency Distribution of Themes Following Coding of All Three Data Sources
Participant
Perception
of
Professional
Learning
Focusing on
Students
Needs and
Instructional
Goals
Individual
Differences
Navigating a
Professional
Development
Website
Evaluating
Information
Encoding
Information
for Future
Retrieval
Continued
Professional
Learning
Intentions for
Practice
Reflecting on
Learning
V01 14 7 4 13 15 3 7 3 17
V02 6 7 12 3 12 12 3 22 10
V03 13 13 6 5 15 11 10 7 21
V04 31 4 5 7 10 1 6 7 18
V05 13 9 4 11 7 1 7 7 14
V06 30 8 5 2 13 1 9 1 8
V07 25 5 6 15 8 11 8 15 19
V08 24 5 7 15 13 1 6 4 29
V09 25 9 5 7 16 5 10 16 13
V10 14 5 7 13 18 5 7 3 17
V11 21 10 7 4 9 4 11 16 11
V12 26 14 5 11 13 10 13 6 25
V13 21 13 9 8 22 4 8 6 15
V14 11 10 9 9 11 1 10 6 16
V15 18 16 6 4 20 9 10 8 15
Total 292 135 97 127 202 79 125 127 248
213
Appendix C-6
Samples from Audit Trail
Raw Data
Think Aloud Transcript (Sample)
V07
January 8, 2014
Transcript
Okay, the first thing I was looking at was who the author of the website was, and so I
clicked on the link at the bottom and I saw that it was OISE and since I’m familiar with that
institution I didn’t feel like I needed to explore that website anymore. And then I went up
and started to read about the Melissa Institute website, and then I wanted to learn a little bit
more about it by reading what was on the main page.
And then I was interested in who was sort of behind the institution so I clicked on the about
us button and read a little bit about the history about why the institution was set up. And
then I scrolled down to read about the members of the board. And at this point what I was
also looking at, I was hoping to find something about how the institute was funded, but
given my time constrain I didn’t want to spend too much time doing that. So I just quickly
looked at who was on the board of directors and so I wanted to see if there was sort of a
balance of educators and I was noticing that there were lawyers and people from the
medical scientific community and just wanted to keep that in the back of my mind in terms
of the credibility of the website...
214
Interview Transcript
V09
March 12, 2014
Interview
I: What were your general feelings while you explored the Balanced Literacy Diet website?
V09: I think it was really well developed. I liked all the visuals, the fact that there’s some video
included for how to present concepts to students. But also the bulletins and how to put it on
display for students and what came out of their work. That really grabbed my attention and
images always seems to work rather than looking at a whole bunch of writing. For me it was
really broken down really well too, but the image really had to capture me to look at the reading.
I: On a scale of 1 to 7, 1 indicating very uncomfortable and 7 indicating very comfortable, how
did you feel about thinking aloud while you watched a screen recording of your navigation?
V09: Probably a 6, I was pretty comfortable. At certain times I was trying to figure out where I
was going. In my mind I was trying to figure out the website initially and I was all over the place
so I wasn’t really sure what I was thinking. I was just trying to get the concept and get to an
image and get to something new. And sometimes I would go back and it would be messy as to
where to click next but I mean I’m not the smoothest when it comes to transitioning on the
computer with something new. But overall I think it was all right cause I did get through a lot of
it. I did do a little repeating of it though.
I: What did you find challenging while you explored the Balanced Literacy Diet website?
V09: Well like I said initially food groups to me, that part, that title, it just kept bringing me back
to health, but then when I clicked on it, it was very clear that they…a couple times I looked at it
and I wasn’t sure what the tab was about and so I went back to it but overall it was well
organized and it made sense and there were lots of drop down menus to go to.
I: What features of the Balanced Literacy Diet website did you find most appealing?
V09: The ideas and the display ideas and the presentation ideas for the students. The talking
through videos up to a point but I don’t spend as much time on that as I do normally. I do like
sometimes if it’s a lesson that I’m really interested in doing it’s nice to see it modeled. It’s also
nice to see it explained out in writing but the modeling actually gives it a good idea of what the
idea would look like… and I think normally when I look at sites like this, if I see something that
interests me I would be jotting down little notes to remind myself so poster-wise, key concepts
and how it was laid out so I can copy that at some point.
I: What other features would you have liked to view/explore in the Balanced Literacy Diet
website?
V09: I guess something that might have been nice is to have some sort of activities linked to it.
So I don’t know that this site had it but if there’s a concept that’s trying to be taught to try and
even have an interactive game that students can then use on the smart board. I didn’t get the
sense that there was a lot of that or any on this site. Or even the links, I didn’t look at any of the
links cause I didn’t want to leave the website, but even if it could link up to other smart board
activities I think.
I: Is there anything else you would like to share about online professional resources?
V09: I guess the one frustration with using the internet is the computer system in the school
system are pretty slow and not terribly well designed. We have a lot of problems at our school
with technology so, it’s not the most reliable of getting information to the students. You can plan
a whole lesson and then the smart board might be taken out for a staff meeting or it might
disappear somewhere else and it becomes a bit of a problem and so what we did have previously
is one smart board traveling to all the grade classes. So we have sixteen grade two classes and
215
one smart board would rotate around to the different classrooms and you would get it for a
couple weeks. But just being ready for it and having the lessons and timing it with the program
and trying to get through especially at the beginning of the year and report card time, people just
want to get through material. Most teachers don’t want the smart board in their room taking up
space…it’s not always the most effective way of teaching. And the speed at school makes it
frustrating. So I do a lot more research on websites at home in my own time.
I: Is there anything else you would like to share about your experience of thinking aloud while
exploring The Balanced Literacy Diet website?
V09: I tend to think aloud a lot even when there’s no one around so it’s not that difficult I think.
216
Screen Shot of Participant Screen Recording
217
NVivo Screen Shots
Sample of Nine Major Themes and Sub-Themes
Sample of Coding Process
218
Theory Construction
Model 1
Model 2
219
Model 3
Causal Conditions
Beliefs about
professional learning
and literacy
instruction
Focusing on Student
Needs and Goals
Individual Differences
Central Phenomenon
Navigating a Professional
Development Website
Strategies
Analyzing and
Evaluating
Encoding Information
for Future Retrieval
Potential Outcomes
Reflecting on Learning
Continued Professional
Learning
Intentions for Practice
affect leads to
brings about
Elementary Teachers’ Learning Experiences as they
Navigated a Professional Development Literacy Website
220
Final Model
Elementary Teachers’ Learning Experiences as they Navigate a
Professional Development Website
Conditions Affecting Central Phenomenon Strategies for Navigating
Navigation a Professional Development
Website
Theme 1:
Perception of
Professional Learning
Theme 4:
Navigating a
Professional
Development Website
Theme 5:
Evaluating
Information
Theme 2:
Focusing on Student
Needs and Instructional
Goals
Theme 6:
Encoding
Information for
Future Retrieval
Theme 3:
Individual
Differences
Potential Outcomes of Navigating a Professional Development Website
Theme 7:
Reflecting on Learning
Theme 8:
Continued Professional
Learning
Theme 9:
Intentions for Practice