Collaborative Group Work: How Students

Feel About the ProcessRobert Gervey

Chia-Chiang Wang Mary O'Connor Drout

Department of Rehabilitation Psychology & Special Education

University of Wisconsin-Madison

Study Setting RP&SE 500

Foundations of Rehabilitation Counseling An entry-level course in the Master’s program for

rehabilitation counselors. It teaches: Legislative history of profession Trends in societal attitudes and behaviors toward

individuals with disabilities Ethical issues of the profession Vocational rehabilitation systems and practices Roles and functions of rehabilitation counselor

Structure of the Course: Pre-Collaborative Group Work

Class has been structured for past several years as a: Hybrid in-class/web-based course Mix of guest speakers, didactic presentations, small group dis

cussions and five in-class debates Five debate topics

Is history an inevitable story of progress and improvement? Has the Americans with Disabilities Act been successful? Should a person who makes bad life choices be eligible for lim

ited federal-state vocational rehabilitation funding? Should the feeding tube have been removed from Terri Schiav

o? Should sheltered workshops be closed?

Rationale for Introducing Collaborative Group Work Exercises To encourage students to provide constructive criticism t

o one another so as to enhance Each participant’s level of participation in debates Each participant’s quality of participation in debates The overall team performance of the debates

To improve self-reflection and self-appraisal about group collaboration skills

To provide students with effective strategies in working with a diverse group under a high stress (time limited), high demand (grade involved) situation that approximates case planning meetings and/or multi-agency, multi-disciplinary case treatment conferences

Sample 27 students

5 master students 22 undergraduates

Gender Male-5 Female-22

12 report prior experience participating in classroom debates

Method: Assignment to Collaborative Work Groups Developed 4 teams so as to create small

groups of 6-7 students to collaborate in 5 class debates Randomly assigned students to collaborative work

groups Group membership remained constant throughout

semester

Method: Structure of Debates Debates were administered in two separate classroo

ms led by the Instructor or the TA Individual teams rotated so that each team had the

opportunity to debate each other team and also be evaluated equally by the TA and Instructor

Each team member had 5 minutes to present their initial argument and 2 minutes for rebuttal

Instructor/TA and 1 or 2 members of each team served as judges for the debate

Collaboration: Defined Week prior to each class debate

On-line threaded group discussion was mandated during which time students were to expected to plan, critique and refine their arguments for the upcoming in-class debate. Students needed to: Post early and often Comment on others postings Edit/modify their arguments based on comments fro

m others Submit written argument

document in dropbox prior to debate

Collaborative Group Work Evaluations Self and Peer Assessments

Students completed on-line (Learn at UW) assessment of collaborative group members’ behavior during week preceding class debate (5 separate assessments-one after each of the 5 debates)

Class evaluation Students completed in-class paper-pencil self report of

learning experience of just completed week of collaborative group work preparation and debate (5 separate evaluations- one after each of the 5 debates)

Course evaluation Students completed pre-post in-class course evaluation

Collaborative Group Work Monitoring: Fidelity

Instructor/TA Monitoring and Prompts Instructor and TA comments made within the

threaded discussions to prompt, probe and direct participation and guide argument formation and to provide constructive feedback about specific types of collaborative group work activity evidenced by group members

Collaborative Behavior: Feedback Report Card includes:

Self- and Peer-Assessment of Collaborative Behavior During Week Preceding Class Debate

Number of Postings Made Number of Posting Read Instructor grade for in-class debate argument Instructor grade for written argument submitted to Dropbox Peer written feedback about Collaborative Behavior across

3 Weeks of Collaboration In-Class, Small Group, Face-to-Face Verbal Feedba

ck from and to Collaborative Group Members (no Instructor or TA present)

Outcome Evaluations: Instructor Generated Verbal Performance

Instructor/TA ratings of individual in-class debate arguments (0-2 scale) (10-15 minute per student)

Written Performance Instructor/TA ratings of individual debate argument submitted to L

earn at UW Dropbox prior to debate (0-2 scale) (1-2 pages in length)

Participation Number/Percentage of Postings within Group Threaded Discussi

on Number/Percentage of Postings Read Timeliness of Postings (Latency of Response)

Results

Enjoyed collaborative debate activity?

I enjoyed the collaborative debate activity

0.0%20.0%40.0%60.0%80.0%

100.0%

Pre Debate 1 Debate 2 Debate 3 Debate 4 Post

Agreement

Increased understanding of materials?

Working in a group increased understanding of material

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Pre Debate 1 Debate 2 Debate 3 Debate 4 Post

Agreement

Helped understand the multiple perspectives on

the issue?Working in a group helped me to understand the multiple

perspectives on the issue

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Pre Debate 1 Debate 2 Debate 3 Debate 4 Post

Agreement

Like the idea of rate self and teammates in terms of collaboration process?

I like the idea of rating myself and my teammates in termsof our collaboration process

0.0%20.0%40.0%60.0%80.0%

100.0%

Debate 1 Debate 2 Debate 3 Debate 4 Post

Agreement

Feedback from teammates enhanced my writing and the strength of my debate?

The feedback from my group members enhanced mywriting and the strength of my debate

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Debate 1 Debate 2 Debate 3 Debate 4

Agreement

On-line collaborative process more efficient and productive than face-face?

I found on-line collaborative group process more efficientand productive than having to meet my team face-face

0.0%20.0%40.0%60.0%80.0%

100.0%

Debate 1 Debate 2 Debate 3 Debate 4 Post

Agreement

Level of comfort

Level of comfort with the teaching methods used in thiscourse

0.0%

20.0%

40.0%60.0%

80.0%

100.0%

Debate 1 Debate 2 Debate 3 Debate 4 Post

Level of comfort

Post: 62% in 2008 vs. 85% in 2007

Level of satisfaction

level of satisfaction with the teaching methods used in thiscourse

0.0%20.0%40.0%60.0%80.0%

100.0%

Debate 1 Debate 2 Debate 3 Debate 4 Post

level of satisfaction

Post: 54% in 2008 vs. 90% in 2007

Examples of Qualitative Assessment: Collaborative Work Group Member Feedback to Alice I appreciate your hard work and effort in the group projects. I

believe the group would benefit from an increased confidence in your work.

Alice is generally very timely with her discussion posts and is very good about commenting on others’ write-ups.

I feel she has shown to be very punctual and precise in her arguments. She brings a relaxed feel to the group which I appreciate. I feel she’s on the higher end in terms of postings and feedback.

Examples of Qualitative Assessment: Alice continued I thought you did a good job helping to organiz

e our group and you gave good feedback. Way to be on top of things!

You are always great about reading everyone’s write ups and giving constructive feedback. Very encouraging and helpful and willing to put in the extra work. Perhaps could post a little bit earlier.

Qualitative Assessments Related to Leadership: Examples of Statements Made to Different Individuals I think you’re a really great leader! You do a really good job of

posting early and providing lots of really helpful feedback for other group members.

I think your leadership is great – I would suggest using this leadership to give more feedback on discussion boards.

You take leadership where necessary. I like that you aren’t overwhelming and you get the job done.

______ did an excellent job starting off our group when no one knew exactly what was expected.

Qualitative Assessment: Organization The only suggestion I can think of might be to work on

organizing your arguments a little more: You usually find really good points and good data for backing up what you’re saying, but sometimes I have a little trouble following your train of thought.

I would like to see more organization in what you say in the

actual debate. It always seems you have a lot of great ideas, but they are not broken down into simple arguments to follow

Qualitative Assessment: Strength of Argument I feel like you really understand the debate materials and

topics and you always come up with really solid arguments. I have also noticed a few times where your arguments have

kind of seemed to summarize the points other people included in their arguments, and I feel like this is somewhat unfair to some of the other members who worked hard on their own, original arguments.

I appreciate that your debates are strong and that you do your research.

Qualitative Assessment-Timeliness

You’re really good at getting your debate points posted early

You are fantastic at very timely feedback on-line and that is very helpful when it comes to tightening up our points.

I know things are really busy and everyone’s schedules are hectic, but sometimes I feel a little bothered by how late you post your debate points, particularly if you’re asking for feedback.

Although I understand you have a very busy schedule it was difficult when you couldn’t’ make the debate on time.

Qualitative Assessment-Feedback

You seem to provide a lot of helpful feedback to other group members.

Good feedback but I think she could be a little more assertive when she has an idea

You provided some great constructive feedback

I think maybe you could provide our team members with a little more constructive feedback rather than just the blanket statement of “good job”.

Qualitative Assessment-Feedback

You’re really responsive to other people’s feedback to your arguments.

I would have liked to see more discussion from you online, and I think you, along with the rest of us, could have benefited from incorporating some feedback into your arguments.

She was helpful to others and not afraid to ask for help herself, which made me feel better about my own confusion with the material.

She did not hesitate to revised her debates based on our feedback

Other Student Comments Let someone else have the first word for once. I

admire that you get your work done quickly, but others are slower at reading and response.

I think your participation, responses, and leadership were all on the right level exactly! I’d love it if you could encourage and support some of the other group members to do the same!

I think it would be great to see your increased confidence and initiative in these projects.

Although you have great input, maybe holding off and letting others post first might be beneficial to the group.

Other Student Comments You seem like a really hard worker and I like the way you

really get involved in the debates. I think she always has a positive attitude and willing to take

on sections or arguments (when in the debate) that are challenging.

I am pleased with the rotation of duties. I see great potential in your thoughts! You do a really great job and I think you’re a really strong

team member.

Student Comments—Written vs. Verbal Feedback “Thanks for allowing us to have time to hash

this all out face to face. I will feel much more comfortable presenting my constructive criticism in person. This will also provide an opportunity to collaborate as a team to stratagize (sic) ways in which we can support one another in achieving these suggestions.”

Students’ Assessment about Assessment “I don’t really have anything to say about

anyone specific. I appreciate those who are able to lead our group and begin the discussions.”

“Overall, I am very pleased with my group and their effort. I know that this is not necessarily what Dr. Gervey was looking for; however, if there is nothing to say, why make up something?”

Self- vs. Peer- Assessment

Pattern for rating Self vs. Peers Below: 6 (23%) Even: 11 (42%) Above: 2 (8%) No Pattern: 7 (27%)

+/- 2.5 points allowed

Self-rating = Peer-ratings Debate 1: 11 (42%) Debate 2: 15 (60%) Debate 3: 13 (52%) Debate 4: 11 (44%)

+/- 2.5 points allowed

•62% of students (N=16) agreed that s/he was tougher when rating self than when rating teammates

•Subjective data

Post- Survey The requirement of having to give feedback to

my group members was helpful 42% (N=11) for agreement

Written vs. Verbal feedback from my group members enhanced my class performance Written 58% vs. Verbal Verbal 73%73%

I would like other courses that I attend to use collaborative projects as a learning method 54% (N=14) for agreement

Relationship between Process and Performance: Where is the Beef? Peer ratings of group collaboration and

instructor generated performance ratings Inconsistent findings

Significant correlations found only with Debate 2 (r = .435, p <.05) Debate 3 (r = .601, p <.01)

To Collaborate or Not: Students’ Choice For final project, students were given a

choice to collaborate Only 6 of 27 elected to work collaboratively Reasons given for working independently

52% - more flexible and time-saving 17% - the site I assessed near my home 13% - learn more by doing it solo 9% - hard to co-work with others 9% - other

Discussion

Did we actually make the student experience of the class worse by adding the collaborative group work process ?

Students’ overall satisfaction with course dropped significantly from year prior

Students voiced quite a bit of displeasure with collaborative group work process Students did not particularly appreciate quantitative report

card style feedback- reported too many numbers andstatistics

Discussion Are we over-estimating the enjoyment and utility of In

ternet in terms of replacing or augmenting in-class, face-to-face interactions? Strong student preference voiced for face-to-face verbal fee

dback Did we select the correct activity to teach and measu

re collaboration? Debates appeared to be perfectly suited

The frequency and various methods of assessment seemed excessive for both students and Instructor/TA Data overload Diminishing returns?

Implications/Future Research Questions Do we need more upfront discussion about students

’ pre-class attitudes and experiences concerning collaborative group work?

Should we expect collaborative group work exercises to yield the results that we had anticipated Enhanced class participation Enhanced quality of work Enhanced match between self and peer assessment of coll

aborative group work skills Providing students with an individual and group grad

e for their collaborative group work does not appear to resolve grading concerns

Limitations

Possible confounds Instructor/TA differences between years Cohort effect- different set of students Was it the debate or the collaborative group work

activity that was the source of displeasure Rating instruments and methods not tested

for reliability or validity

Questions?? Comments??

Thank you

Special thanks go to DOIT and ENGAGE for providing all needed consultation and technical supports.