Using Portfolio Assessment to Discover Student

LearningBarbara J. D’Angelo, Ph.D.Arizona State University

Association for Business Communication Annual Convention

November 2009

Background: Written Communication for Managers Junior-level business writing course

Multi-section course 7-10 sections per semester + summer On-campus, hybrid, and online

Redesigned Summer 2007 Industry focus groups Business and writing faculty input Role playing scenario: students create a company Standardized syllabus and assignments

Portfolio for assessment End-of-semester self-evaluation of learning Scored by 1 writing faculty and 1 business faculty

Barbara J. D'Angelo, ABC Convention November 2009 2

Research QuestionsWhich assignments do students choose to

include in their portfolios?

How do students organize their portfolios as an argument to demonstrate their learning?

What persuasive techniques do students use to support their argument for learning?

What tone and style do students use?

Do students present an argument for their learning or to enhance their grade?

Barbara J. D'Angelo, ABC Convention November 2009 3

MethodsRandom sampling

3 portfolios from each section Fall 2008 and Spring 2009 34 portfolios analyzed

Simple counting of assignments

Content analysis of narrative statements Categories derived from

Bower (2003): rhetorical appeals and argumentationScott (2005): form, tone, and appeal

Post-semester feedback from instructors

See proceedings for details (or talk to me later or email me)

Barbara J. D'Angelo, ABC Convention November 2009 4

Assignments Included in Portfolios

Barbara J. D'Angelo, ABC Convention November 2009 5

Fall 2008 Spring 2009

Memo 11 14

Manager Roundtable

10 12

Proposal 10 10

“Branding” 9 11

Letter 8 14

Policy 9 10

Audience Analysis 6 1

Agenda 4 7

Email 4 5

Peer Review 4 5

Research Summary 3 3

Minutes 2 7

AssignmentsAssignment inclusion not related to grading scale

Inclusion seems based on Substantiality of writing or traditional genres Instructor emphasis: instructor feedback indicates this

as high

De-emphasis of meeting documents Lack of understanding or emphasis Increased emphasis during spring = higher inclusion =

awareness of importance?

Process documents Reluctance to show drafts or other “in progress” work?

6Barbara J. D'Angelo, ABC Convention November 2009

Portfolio Organization

26 organized generically Introduction: reflection about courseBody: described artifacts, claims and evidenceConclusion: summed up

7 organized contextuallyBody contained more context

Best vs. worstStrengths vs. weaknessesCourse objectives

Had no influence on raters’ scores

Barbara J. D'Angelo, ABC Convention November 2009 7

Persuasive techniques# of

Portfolios# of

Statements

Ethos 14 19

Logos 23 51

Pathos 17 29

Nomos 21 31

Skills Supported 23 64

Skills Unsupported 18 39

Barbara J. D'Angelo, ABC Convention November 2009 8

Use of PersuasionNomos and pathos not linked to “schmooze”

Not directly linked to a claim or evidence

More expression of student “feelings”

Awareness of audience and course values

Majority (23) of students supported claims 23 used logos to create claim with evidence to support it

13 supported all claimsUnlike findings with FYC students

Indication of more sophisticated writing from upper level students?

8 did not support any claims

9Barbara J. D'Angelo, ABC Convention November 2009

Tone and Style 24 students used positive tone

Excitement about role playing (pathos)

3 students used negative toneDis-satisfied with grade or with instructor?

7 students used intermediate toneNeutral descriptive tone

All students changed from “business” style to “student” style when completing the portfolio

Barbara J. D'Angelo, ABC Convention November 2009 10

Learning or enhance grade?

Learning 28 students made statements related to their growth Only 1 student directly referred to their grade Use of logos and support for claims indicates argument

was for learning

Enhance grade 19 portfolios potentially included “schmooze”

2 flattered course or instructorRest thanked instructor

2 students appealed directly to instructor (using “you”)

Result differs from FYC studies

Barbara J. D'Angelo, ABC Convention November 2009 11

Use for course improvement More emphasis on meeting documents

Importance to our employer stakeholders

Additional requirement to write agendas added for each Manager Roundtable

Requirement to evaluate agendas

Change MRT topics to more completely support written assignments Instructors believed not enough time to go over concepts

needed for assignments

Strengthen research summary to emphasize tie between finding information-analysis-use for persuasion

12Barbara J. D'Angelo, ABC Convention November 2009

FuturePotential change: portfolio scenario to shareholder

or other external audience Maintain emphasis on business style

Continue to de-emphasize mechanicsBut help needed for ESL students

Continue assessment to aggregate additional data

Confirmation of findings related to use of logos/evidence

Add industry evaluators to assessment to include external stakeholders

13Barbara J. D'Angelo, ABC Convention November 2009

Questions?Presentation available at:

http://www.public.asu.edu/~bdangelo/presentations/abc09.ppt

Course syllabus and assignments available at:

http://techcomm.asu.edu/curriculum/twc347

Email: bdangelo@asu.edu

Barbara J. D'Angelo, ABC Convention November 2009 14