BRIEF REPORT

Exploring the Process of Planning and Implementation Phasesin an Online Project-Based Discussion Activity Integratinga Collaborative Concept-Mapping Tool

Sheng-Yi Wu • Huei-Tse Hou

� De La Salle University 2013

Abstract Project-based learning may enhance students’

cognitive skills and knowledge construction. Online dis-

cussion stimulates the social interactions of project-based

learning, and appropriate cognitive tools (such as concept

maps) facilitate the coordination, planning, and implemen-

tation of projects. Currently, related studies on project-based

learning environments that integrate concept-mapping and

discussion tools are sporadic. This study explored the

behavioral patterns of learners’ concept-mapping processes

and the cognitive processing patterns of online discussion

content conducted in a project-based learning environment

equipped with both concept-mapping and online discussion

tools. This study also compared the differences in behavioral

patterns in the two main project-based learning phases:

planning and implementation. This study employed content

and sequential analyses to conduct empirical analyses with

48 college students as participants over a two-month period

of teaching activities. The results showed that the behavioral

patterns of learners exhibited more diversified operations

and discussion during the planning phase. In addition, the

use of concept mapping allowed participants to focus on the

discussion task. This study proposes teaching practices and

research recommendations based on the results.

Keywords Behavioral pattern � Cognitive processing �Concept mapping � Online discussion �Project-based learning

Introduction

Project-based learning (Polman 2000) encourages collab-

oration and communication among students as they con-

struct their knowledge and complete a project during the

learning process. Through project-based learning, teachers

can cultivate students’ abilities in project management,

organization, expression, reflection, and teamwork, and

they can help students solve real-life problems (Thomas

2000). Therefore, through the learning process, project-

based learning can enhance students’ cognitive thinking

and knowledge construction, and it can provide practical

problem-solving scenarios (Krajcik et al. 2003). With the

rapid development of internet technology, online project-

based learning has become a widespread teaching strategy

that is adopted by many researchers and teachers (Hou

2010).

Among many online project-based learning activities,

one that is often used to assist students is combining pro-

ject-based tasks with online discussion activities. Online

project-based discussion activities make the teaching pro-

cess more interactive and increase students’ in-depth

understanding of the discussion topic. Discussion activities

also promote students’ collaborative learning, cognitive

skills, and critical thinking abilities (Anderson et al. 2001).

However, despite their many positive effects, studies have

noted that when online discussion activities are conducted

without the intervention of teachers or other system tools,

there are many limitations (problems such as defocusing

and off-topic discussions) (Walshaw and Anthony 2008;

S.-Y. Wu

Aim for the Top University Project Office, National Taiwan

Normal University, #162, Heping East Road Section 1,

106 Taipei, Taiwan

e-mail: digschool@gmail.com

H.-T. Hou (&)

Graduate Institute of Applied Science and Technology,

National Taiwan University of Science and Technology,

#43 Keelung Road, Section 4, Taipei, Taiwan

e-mail: hthou@mail.ntust.edu.tw

123

Asia-Pacific Edu Res

DOI 10.1007/s40299-013-0089-6

Hou and Wu 2011). As a solution, some studies have

suggested conducting online discussion activities using

cognitive tools. A concept map is commonly regarded as a

cognitive tool that provides a graphical strategy for orga-

nizing various concepts through a topical architecture

(Novak and Gowin 1984) so that the concepts, often

expressed in nodes, are connected with labeled arrows to

show their relationship. Concept mapping can efficiently

achieve knowledge retention and transfer (Nesbit and

Adesope 2006). Therefore, the combination of concept

mapping and online discussion may have the potential to

offset the limitations of project-based discussion activities.

However, tools that integrate collaborative concept

mapping with online discussion are uncommon, and empir-

ical studies are insufficient. Project-based discussion

activities consist of two major phases, planning and imple-

mentation (Thomas 2000), so the subsidiary effectiveness

and limitations of such tools can vary according to the

characteristics of each phase during instructional practices.

Therefore, understanding students’ behavioral patterns in

concept-mapping operations and their cognitive processes

during discussion within the collaborative concept-mapping

online discussion environment can help teachers better uti-

lize this environment and can offer important and useful

practical references for the design of different phases of

project-based activities. There are currently no such studies.

To thoroughly comprehend the operating behavioral

patterns and cognitive processes of the concept-mapping

online discussion-teaching environment, the use of an

integrated quantitative content analysis of the discussion

content and a lag sequential analysis of the operating

behavior can facilitate understanding of the operating

behavioral patterns and cognitive processes (e.g., Hou and

Wu 2011; Wu et al. 2012). The lag sequential analysis aims

to discover the sequential patterns in a stream of coding

categories that describe interactions (Bakeman and Gott-

man 1997) and to visualize behavioral patterns. Many

researchers have applied this method of analysis to

behavioral patterns in the fields of education, science, and

technology, such as students’ group interactions in online

threaded discussions (Jeong 2003) and learners’ behavioral

patterns in an educational massively multiplayer online

role-playing game (Hou 2012).

This study aimed to integrate concept mapping with

online discussion to create a collaborative online learning

environment. In other words, students needed to engage in

the process of collaboration during the project task (Dillen-

bourg 1999) and collaboratively plan and implement a pro-

ject. This environment was used to explore behavioral

characteristics and differences in the planning and imple-

mentation phases of students’ online project-based learning

activities through quantitative content analysis and empirical

sequential analysis. The research questions are as follows:

1. What are the differences in students’ behavioral

patterns when operating concept-mapping tools during

the planning and implementation phases of project-

based learning activities?

2. What are the differences in students’ discussion

content and cognitive processes during the planning

and implementation phases of project-based learning

activities?

Methods

Participants and Procedures

The participants in this study, 48 freshmen (41 girls and 7

boys with an average age of 19) taking the course ‘‘Intro-

duction to Digital Content,’’ were divided into 12 groups of

four. All participating students were assessed prior to the

experiment and had comparable information technology

abilities and experiences in online discussions. We adopted

the collaborative concept-mapping system (CCMS) devel-

oped by our research team for the online discus-sion activi-

ties. Each group conducted online discus-sions through the

CCMS and completed the planning and implementation of a

multimedia project within 2 months (planning and imple-

mentation each took 1 month). Finally, each group submit-

ted the plan and the physical presentation of their multimedia

project. In the planning phase of the multimedia project,

students needed to collaboratively plan the contents of a

multimedia project via discussion and present the framework

of the project via concept mapping. In the implementation

phase of a multimedia project, the groups needed to discuss

how to conceive and produce a multimedia project via the

CCMS and pres-ent the work coordination and procedures

via concept mapping.

Collaborative Concept-Mapping System (CCMS)

The CCMS was developed by our research team. It is an

online system that allows synchronous multiple online

discussions and the sketching of collaborative concept

maps. When conducting online discussions and sketching

concept maps, students’ dialogues and their system oper-

ating behavior are recorded and used for post-hoc analysis

(Fig. 1).

Coding Schemes

To understand the behavioral patterns and cognitive pro-

cesses, the study used two types of coding schemes. To

analyze the system operating behavioral patterns of the first

S. Wu, H. Hou

123

research question, this study introduced the CCMS behav-

ior-coding scheme (C code) based on the system charac-

teristics. This scheme contains eight types of behavior, add a

node (AN), delete a node (DN), move a node (MN), rename

name of node (RN), add a relationship link (AL), delete a

relationship link (DL), rename name of relationship link

(RL), and talk to peers (T). To answer the second research

question, this study applied the revised Bloom’s taxonomy

(Anderson 2006) to understand the cognitive processing

phases (B code). It included remember (B1), understand

(B2), apply (B3), analyze (B4), evaluate (B5), and create

(B6). In addition, during the process of discussion, all con-

versations unrelated to the learning topic were classified as

off-topic (B7).

Data Analysis

After the completion of the observation, the system

operating behaviors and discussion content were coded.

The system operating behaviors were coded directly from

the system logs. In contrast, the discussion content was

coded with the unit of each message that students posted

in the discussion tool (the message probably included

paragraphs composed of many sentences). The messages

were first coded based on the cognitive processing phases

by two coders with psychology backgrounds, and the

inter-rater Kappa test was then used to confirm the reli-

ability of the coding. After the coding was complete, the

researchers analyzed the frequency and proportion dis-

tribution of the codes and conducted a lag sequential

analysis. The analytical methods included sorting the

aforementioned codes based on time and then calculating

the frequency transfer matrix. Through a series of calcu-

lations for the behavior transfer matrix, each sequence

that achieved continuous significance can be inferred

(refer to Bakeman and Gottman 1997; Hou 2010). Based

on the data, the sequential transfer diagrams were drawn

to understand the sequential behavioral patterns of stu-

dents’ cognitive phases and CCMS behaviors.

Results and Discussion

Table 1 presents the frequency of each behavior in the

planning and implementation phases; the results show that

in the behavior-coding scheme (C code), there are small

differences in the frequency of each code between the

planning phase and the implementation phase. However, in

the cognitive processing phases (B code), the frequency of

each code in the planning phase is higher than that in the

implementation phase, indicating that students have more

discussions during the implementation phase.

Analysis of Differences in Behavioral Patterns

of Concept-Mapping Tool Operation During

the Planning and Implementation Phases

Tables 2 and 3 show the frequency of the behavioral transfers

in the planning phase and the implementation phase,

respectively. The rows represent the starting behaviors, and

Fig. 1 CCMS system a add a concept map, nodes and relationships, b group members and permission change, c discussion board, d area

illustrating the concept map)

Process of Planning and Implementation Phases in an Online Project

123

the columns represent the behavior immediately following

the starting behaviors. For example, in the planning phase,

the frequency of understand (B2) to off-topic (B7) is 210.

Following the sequential analysis of a series of behavioral

transfer matrixes (Bakeman and Gottman 1997), the system

operating behavior sequences (C Code) of the planning phase

and the implementation phase that achieved statistical sig-

nificance (p \ 0.05) are shown in Figs. 2 and 3, respectively.

In these figures, ‘‘?’’ indicates the direction of the significant

sequence.

As seen in Fig. 2, the behaviors that yielded the largest

difference in the planning and implementation phases were

the significant reciprocal sequences between rename rela-

tionship links (RL) and add relationship links (AL) and

between RL and delete relationship links (DL) (i.e.,

AL ? RL, RL ? DL). This means that during the plan-

ning phase, students exhibited repeated behavior of adding,

modifying, or deleting relationship links. In other words,

while planning their projects, students would often change

relationship links on the concept map, possibly due to the

multiple views presented in their discussion. Compared to

the planning phase, the RL behavior did not occur.

Therefore, there were fewer occurrences of modifying

relationship links during the implementation phase because

there may have been few differences of opinion. This

finding suggests the need for more diverse thinking and

coordination between group members during the planning

phase, which would subsequently generate more diversified

behavioral transfer patterns in the operation of concept-

mapping tools. In other words, using concept maps in the

planning phase should benefit the communication and

coordination between group members.

Analysis of Differences of Cognitive Processes

in the Discussion Content During the Planning

and Implementation Phases

After a total of 3,337 discussion messages were coded over

a period of 2 months, the inter-rater Kappa reliability

coefficients between the two coders was 0.852 (p \ 0.001),

indicating a very high level of consistency. Through cate-

gorizing and coding T code in C code (i.e., peer-discussion)

based on the cognitive processing phases (B code), this

study conducted a lag sequential analysis by integrating the

Table 1 Behavior frequency of

the planning and

implementation phases

Codes Planning phases Implementation phases

B2 448 100

B4 10 0

B6 156 18

B7 1768 940

Total 2,382 1,058

AL 604 582

AN 648 616

DL 211 216

DN 265 229

MN 3,682 3,785

RL 1 0

RN 94 81

Total 5,505 5,509

Table 2 Behavioral transfer

frequency of the planning phaseX B2 B4 B6 B7 AL AN DL DN MN RL RN

B2 145 2 32 210 7 20 0 5 24 0 3

B4 1 2 1 4 1 0 0 0 1 0 0

B6 34 1 21 67 8 15 0 2 7 0 1

B7 209 4 76 1,222 34 68 2 24 111 0 11

AL 7 0 6 46 298 57 1 17 160 1 8

AN 12 0 6 33 92 285 0 19 193 0 8

DL 1 0 0 5 5 41 95 42 21 0 1

DN 4 0 0 15 5 19 112 92 18 0 0

MN 33 1 13 145 146 133 0 59 3,118 0 32

RL 0 0 0 0 0 0 1 0 0 0 0

RN 2 0 1 9 8 10 0 5 29 0 30

S. Wu, H. Hou

123

B code with the original concept-mapping operating

behavior of the C code. Within these analyses, the signif-

icant sequences between each behavior of the B code were

drawn out. The behavioral transfer diagram of the cognitive

processing phases (B code) during the planning phase and

the implementation phase is shown in Figs. 4 and 5,

respectively.

As shown in Fig. 4, the analyze (B4) behavior appears

in the planning phase. This means that during the planning

phase, group members engaged in the behavior of com-

paring, examining, or testing while discussing and think-

ing. For instance, the group discussion during the planning

phase focused on comparing the suitability and difficulties

of their ideas. However, the analyze (B4) behavior failed to

appear in the implementation phase. This may be because

during the implementation phase, group members needed

only to focus on developing and producing the project

based on the plan and did not wish to engage in any ana-

lytical behavior in the implementation phase. Therefore, it

is evident that applying CCMS in the planning phase may

promote a more diversified behavioral transfer pattern of

cognitive process.

The planning phase was augmented with the significant

sequence of create (B6) to off-topic (B7) (B6 ? B7),

indicating that the create (B6) discussion behavior was

easily transferred to the off-topic (B7) discussion, most

likely because the topics discussed were divergent. This

Table 3 Behavioral transfer

frequency of the

implementation phase

X B2 B6 B7 AL AN DL DN MN RN

B2 43 3 42 1 3 0 1 6 1

B6 7 2 5 0 3 0 0 1 0

B7 35 12 673 23 71 0 14 100 8

AL 1 0 37 339 58 0 15 127 4

AN 1 0 30 74 246 0 10 246 8

DL 1 0 8 1 36 123 31 16 0

DN 0 0 4 2 11 92 100 19 1

MN 12 1 128 138 175 1 56 3,240 29

RN 0 0 6 4 8 0 2 30 30

Fig. 2 Behavioral transfer diagram (C code, the planning phase)

Fig. 3 Behavioral transfer diagram (C code, the implementation

phase)

Fig. 4 Behavioral transfer diagram (B code, the planning phase)

Fig. 5 Behavioral transfer diagram (B code, the implementation

phase)

Process of Planning and Implementation Phases in an Online Project

123

result is similar to the findings of Hou et al. (2007), indi-

cating that learners might have many off-topic behaviors in

a project discussion. However, Fig. 5 also shows that even

if the focus shifted to off-topic (B7), there was still sig-

nificant sequential behavior that moved from off-topic (B7)

back to create (B6). In other words, through the aid of the

CCMS, the group discussion could be guided from off-

topic back to the main topic of discussion. However,

without the aid of the CCMS, Hou et al. (2007) found that

the off-topic behaviors had significant self-continuity and

had the limitation of failing to return to the main topic of

discussion. Therefore, using the CCMS may help teachers

or researchers overcome problems related to learners’ lack

of focus or off-topic discussions in the CSCL context (e.g.,

Walshaw and Anthony 2008; Hou and Wu 2011; Hou et al.

2007).

Implications

To conclude the above two sections, we found that through the

assistance of the CCMS, online discussion activities in the

planning phase consist of more discussion interactions and

more transfer interactions between each behavior than in the

implementation phase. This finding suggests that integrating

concept-mapping tools into online discussion activities is

more appropriate for the planning phase of project-based

discussion activities than it is for the implementation phase.

Compared to past research regarding the adoption of project-

based learning as a learning activity, most existing studies

adopted the overall activity as a unit to conduct a behavior

analysis (e.g., Hou 2010; Hou et al. 2007). However, this

study divided project-based discussion activities into the

planning and implementation phases. The research results

also indicated behavioral differences among students in the

two phases, thus providing important foundations for future

research in this field and in instructional practices. Future

studies should conduct multi-dimensional comparisons and

analyses based on the two phases to give learners different

adaptive guidance based on their behaviors in different

phases.

Conclusions and Suggestions

This study adopted the CCMS and sequential analysis to con-

duct preliminary empirical research and to explore operating

behavioral patterns and differences in cognitive processes

during the planning and implementation phases of online

project-based discussion activities. The study results showed

that learners exhibited more diversified behavioral transfer

patternsofcognitiveprocessesandCCMSoperatingbehaviors

during the planning phase. In addition, cognitive tools con-

tributed to shifting group discussions from off-topic subjects

andguidingthembacktothemaintopicofdiscussion.Thismay

help learners reduce the number of off-topic discussions

and enhance focus in online collaborative discussions (e.g.,

Walshaw and Anthony 2008; Hou and Wu 2011; Hou et al.

2007), thus further promoting learner concentration.

Therefore, in practice, we recommend that teachers use

an integrated concept-mapping online discussion environ-

ment (such as the CCMS) in the future to conduct online

project-based discussion activities, especially in the planning

phase of the project. Regarding future research, this study

found that learners had different behavioral patterns between

the planning phase and the implementation phase during

project-based learning; therefore, we recommend the future

research related to project-based learning to explore the

differences in different dimensions of the two learning

phases more thoroughly, e.g., learning effects and learning

attitudes. In addition, in terms of research methods, this

study adopted a sequential analysis. Future researchers can

also utilize other analytical methods to further explore the

differences and characteristics of the two phases, such as

using social network analysis (Scott 2000) to investigate the

mode of social interaction among team members or applying

progressive sequential analysis (Hou 2010) to explore

dynamic behavioral patterns. Finally, as this study is an

exploratory case study, future related inquiries that include a

larger sample size are recommended.

Acknowledgments This work was supported by ‘‘Aim for the Top

University Project’’ of the National Taiwan Normal University and the

Ministry of Education, Taiwan, R.O.C. and National Science Council,

under contract number NSC-100-2628-S-011-001-MY4, NSC-100-

3113-S-011-001, NSC-100-2631-S-011-002, and NSC -99-2511-

S-011-007-MY3.

References

Anderson, L. (2006). Revised Bloom’s Taxonomy. Paper presented at

North Carolina Career and Technical Education Curriculum

Development Training, Raleigh, NC.

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001).

Assessing teaching presence in a computer conference context.

Journal of Asynchronous Learning Networks, 5(2), 1–17.

Bakeman, R., & Gottman, J. M. (1997). Observing interaction: Anintroduction to sequential analysis (2nd ed.). New York: Cambridge

University Press.

Dillenbourg, P. (Ed.). (1999). Collaborative learning: Cognitive andcomputational approaches.Amsterdam: Pergamon ElsevierScience.

Hou, H. T. (2010). Exploring the behavioural patterns in project-

based learning with online discussion: Quantitative content

analysis and progressive sequential analysis. Turkish OnlineJournal of Educational Technology, 9(3), 52–60.

Hou, H. T. (2012). Analyzing the learning process of an online role-

playing discussion activity. Educational Technology & Society,15(1), 211–222.

Hou, H. T., Chang, K. E., & Sung, Y. T. (2007). An analysis of peer

assessment online discussions within a course that uses project-based

learning. Interactive Learning Environments, 15(3), 237–251.

S. Wu, H. Hou

123

Hou, H. T., & Wu, S. Y. (2011). Analyzing the social knowledge

construction behavioral patterns of an online synchronous

collaborative discussion instructional activity using an instant

messaging tool: A case study. Computers & Education, 57,

1459–1468. doi:10.1016/j.compedu.2011.02.012.

Jeong, A. C. (2003). The sequential analysis of group interaction and

critical thinking in online threaded discussions. American journalof distance education, 17(1), 25–43.

Krajcik, J. S., Czerniak, C., & Berger, C. (2003). Teaching childrenscience in elementary and middle school classrooms: A project-based approach. New York: McGraw-Hill.

Nesbit, J. C., & Adesope, O. O. (2006). Learning with concept and

knowledge maps: A meta-analysis. Review of EducationalResearch, 76, 413–448. doi:10.3102/00346543076003413.

Novak, J. D., & Gowin, D. B. (1984). Learning how to learn.

New York: Cambridge University Press.

Polman, J. L. (2000). Designing project-based science: Connectinglearners through guided inquiry. New York: Teachers College

Press.

Scott, J. (2000). Social network analysis: A handbook. London: SAGE.

Thomas, J. W. (2000). A review of research on project-basedlearning. San Rafael: Autodesk Foundation.

Walshaw, M., & Anthony, G. (2008). The teacher’s role in classroom

discourse: A review of recent research into mathematics classrooms.

Review of Educational Research, 78(3), 516–551. doi:10.3102/

0034654308320292.

Wu, S. Y., Hou, H. T., & Hwang, W. Y. (2012). Exploring students’

cognitive dimensions and behavioral patterns during a synchro-

nous peer assessment discussion activity using instant messag-

ing. The Asia-Pacific Education Researcher, 21(3), 442–453.

Process of Planning and Implementation Phases in an Online Project

123