TYCA Northeast Annual Conference 2010Capitol Improvements: The Two-Year College as an Agency of Change

Collaborative Assessment:

Working Together Toward Institutional Change

Kelly Bender and Elizabeth NesiusPassaic County Community College

This presentation can be found onSlideShare

This presentation looks at methods of collaboration for course-and program-level assessment and data-sharing to help both faculty and program administrators. Instructors can improve the WAC elements of their courses without sacrificing content, while still furthering the program mission of helping students to write well and think critically in any discipline. Writing program administrators, on the other hand, can improve both the writing program implementation and the training of its faculty to teach WAC courses. Students will benefit by experiencing WAC courses that are clearly defined, engaging, and optimized for their learning.

PCCC Writing Initiative

Title V Grant: $2.5 million over 5 yearsDevelop 20 Distinct, Writing Intensive

Courses (with multiple sections, across 3 campuses)

Develop and provide online resources for technology and writing

Support students and faculty with an on-ground writing center and online tutoring

Students required to take 2 WI courses (1 if getting AAS degree)

Writing Intensive Courses

19 Distinct WI courses currently developed or in development across the curriculum

Departments include: Computer Information Systems, English, Humanities, Math, and Science

Courses designed to contain writing, critical thinking, and information literacy elements

What We Assess

Students◦ Assessed in writing, critical thinking, and information

literacy skills◦ Assessed in the course by instructors

Courses◦ Course materials created collaboratively in Faculty

Institutes◦ Course materials checked and approved by WI

AdministratorsProgram

◦ Students assessed across courses by WI administrators in the 3 areas

◦ Students and instructors self-assess by means of a survey

How We Assess

Students◦Instructors: student grades◦Administrators: student performance on CWE

Courses◦Course materials collected at the beginning of each

semester◦Evidence of materials being used by student use of

ePortfoliosProgram

◦Student writing assessed with 3 rubrics: writing, critical thinking, information literacy

◦Student and faculty surveys

Assessing Writing

First we had to determine the criteria for writing assessment:◦Appropriateness◦Unity & Focus◦Development◦Organization◦Sentence Structure◦Usage◦Mechanical Conventions

Additional criteria for course writing are left up to individual instructors

Writing Rubric Student __________________________________ Assignment ______________________________________ Beginning Developing Competent Accomplished

Vocabulary/Language

appropriate to this course

Words used incorrectly; displays fundamental errors in vocabulary usage; lacks variety.

Generally correct usage but little variety; some variety but words used incorrectly.

The majority correctly used with some variety in vocabulary.

Varied vocabulary; correct and sophisticated use of language.

Conventions mechanics, spelling, sentence structure,

word forms (problems noted below)

Many serious errors which persistently interfere with meaning.

Some serious errors that may interfere with meaning.

A few errors which do not interfere with meaning.

No serious errors; few minor errors.

Organization

includes: appropriate length, order (logical, chronological etc.) and

transitions

No clear structure; organizational requirements not met.

Some structure and order evident; minimally, an introduction, body & conclusion present.

Generally well-organized; clear and logical flow of ideas. Fulfills the assignment.

Well-organized with appropriate structure and smooth transitions.

Content

includes addressing the assigned topic

Lacks thesis; incomplete development of evidence and/or minimal relevance to topic.

May lack thesis. Some evidence underdeveloped or off-topic.

Has supporting evidence and examples for thesis.

Strong, convincing evidence to support thesis and argument.

Tense formation subject-verb agreement pronoun usage/agreement word choice/meaning proper modifiers Variety of formations correct construction Spelling Capitalization punctuation

COMMENTS:

WI v10.1

Assessing Writing

EN 205 – Introduction to Literature Grading Rubric

 

Student __________________________________ Assignment ______________________________________

Accomplished Competent Developing Beginning

Vocabulary/Language appropriate to this course

Varied vocabulary; correct and sophisticated use of language.

The majority correctly used with some variety in vocabulary.

Generally correct usage but little variety.

Words used incorrectly; displays fundamental errors in vocabulary usage; lacks variety.

Conventions (problems noted below)

No fundamental errors; few minor errors.

A few errors which do not interfere with meaning.

Some fundamental errors that may interfere with meaning.

Many fundamental errors which persistently interfere with meaning.

Organization includes: appropriate length, order

(logical, chronological etc.) and transitions

Well-organized with appropriate structure and smooth transitions.

Generally well-organized; follows basic assignment guidelines; clear and logical flow of ideas.

Some structure and order evident, attempt at following assignment guidelines; minimally, introduction, body & conclusion.

No clear structure; does not follow assignment guidelines; incoherent.

Content includes addressing the assigned topic

Strong, convincing evidence to support thesis and argument.

Has supporting evidence and examples for thesis.

May lack thesis or conclusion. Evidence underdeveloped or off-topic.

Lacks thesis and/or conclusion. Minimal relevance to topic.

Citation of sources

Uses citations when appropriate and demonstrates mastery of formatting; goes beyond the amount of citation or sources required.

Uses required amount of citations when appropriate and uses proper formatting most of the time.

Uses citations infrequently and shows inconsistent formatting; understands some elements of citation but not others.

Uses little or no citation and displays little knowledge of formatting; clearly cannot differentiate between most of the elements of citation.

USAGE

Tense formation

subject-verb agreement

pronoun usage/agreement

word choice/meaning

proper modifiers SENTENCE CONSTRUCTION

Variety of formations

correct construction MECHANICS

Spelling

Capitalization

punctuation

COMMENTS:

Revised 11/1/2010

Assessing Writing

Faculty use rubric with students for formative writing assessment

Extra space to add content or other elements for assessment based on assignment

WI Administrators use rubric for summative assessment of writing (without additional elements)

TOOLS: The Writing Center

The Writing Center provides support for students at all stages of the writing process, whether it is brainstorming a topic, structuring an essay, or editing and revising.It also offers support to students for their Writing Intensive courses in the forms of face-to-face and small group tutoring/instruction.

The Writing Center

While the Writing Center made the assessment of student writing “easier,” it was also helpful in terms of assessing the course.

Attendance at the Writing Center was made mandatory. This created a direct correlation between the outcome of student writing and the effectiveness of a support tool.

Assessing Critical Thinking

Criteria for critical thinking assessment◦Identifying arguments and counter-arguments◦Assessing validity of both sides◦Drawing conclusions based on those

assessmentsFaculty use rubric with students for

formative critical thinking assessmentAdministrators assess critical thinking

across courses (in writing assignments that require critical thinking) using same rubric

Assessing Critical Thinking

Critical Thinking Rubric STUDENT _________________________________________ Assignment ______________________________

Beginning Developing Competent Accomplished

Evidence

Biased interpretations Insufficient search for

evidence or inappropriate

evidence choices

Misinterprets some evidence

Limited information search

limited source evaluation or attribution

Accurately interprets the majority of evidence

Sufficient search for evidence

Shows some evaluation of sources

Appropriate evidence Accurately interprets the

evidence presented

Arguments

Fails to identify argument

or counter-arguments

Has argument but inadequate identification of relevant counter-arguments.

Identifies relevant argument(s)

including counter-arguments

Identifies pertinent arguments, pro and con

Rebuts counter arguments

Conclusion(s)

Does not reach any conclusion justified by evidence.

Conclusion based on incorrect, irrelevant , or insufficient evidence

Conclusion(s) are reasonably justified or reasonably explained.

Draws thoroughly justified conclusions by appropriate interpretations of evidence

Comments

V4 June 2010

Assess Information Literacy

Criteria for information literacy assessment◦Defines and articulates need for information◦Information retrieval◦Citation of sources◦Evaluation of Sources◦Uses information effectively

Information literacy required in at least one assignment

Assessment by faculty and administrators using the same rubric

Assessing Information Literacy

Information Literacy Rubric Student ________________________ Reader ____________________ Score _____________ Beginning Developing Competent Accomplished Defines and articulates the need for information

No clear statement of purpose; does not associate information need with the assignment

Shows purpose but not clearly articulated; associates information need with the assignment but does not articulate a thesis statement

Purpose is clearly articulated and information need is clearly recognized

Purpose is highly focused and/or nuanced; thesis statement or research question takes multiple viewpoints into account

Information retrieval

Retrieves little or none of the information required for the assignment; or, information retrieved clearly does not meet the information need

Retrieves some of the information required for the assignment; some of the information retrieved may not meet the information need

Retrieves the required amount of information for the assignment and the majority of it meets the information need

Retrieves much more than the required amount to complete the assignment; information retrieved meets the information need; multiple viewpoints are considered

Citation of sources

Uses little or no citation and displays little knowledge of formatting; clearly cannot differentiate between most of the elements of citation

Uses citations infrequently and shows inconsistent formatting; understands some elements of citation but not others

Uses citations when appropriate and uses proper formatting most of the time

Uses citations when appropriate and demonstrates mastery of formatting; goes beyond the amount of citation or sources required

Evaluation of Sources

Little or no recognition of the relevance and quality (authority, currency, objectivity) of information sources

Recognizes the relevance of information sources but has difficulty evaluating their quality

Recognizes both the quality and relevance of information sources

Information sources are highly relevant and evaluation of sources is highly nuanced

Uses information effectively in their writing assignments

Uses little or no useful information from outside sources

Attempts to incorporate information from outside sources; sometimes off-topic

Uses information from outside sources to support the topic; shows some synthesis with own ideas

Use of information from outside sources is highly refined, articulated, and synthesized with own ideas

Comments:

Revised 6/6/08

Assessing the Course

Course assessment is essential in determining:

Both the quality and quantity of learning for students

An appropriate response to their personal development needs

Whether students have acquired the knowledge, skills and competencies consistent with the aims of the course

The effectiveness of the curriculum and faculty development

Methods of Course Assessment

By Instructors◦Faculty institutes◦ePortfolios◦Course Survey

By Administrators◦Course materials◦ePortfolios

Course Assessment: Faculty Institutes

Faculty Administrators

Institute Part 1◦ Overview of Initiative◦ Overview of WI course

components◦ Look at course as it

stands for writing, critical thinking and information literacy elements

◦ Begin development of new materials with rubrics and templates

Institute Part 1◦ Collect current course

materials (syllabus and assignments)

◦ Assist in development of new materials

◦ Collect and file new materials

◦ Survey faculty on success of workshop components

Faculty Institutes

Faculty Administrators

Institute Part 2◦ Some materials tested

in regular section of courses

◦ Materials refined based on in-course testing

◦ More materials developed

◦ Integration of media and other resources

Institute Part 2◦ Discuss piloting of

materials from part 1◦ Assist in refining

materials and development of new materials

◦ Assist in incorporating media, other resources and support services

Assessing the Initiative

The program is assessed for accomplishment of its goals◦For faculty development◦For student development

Methods of Program Assessment◦ePortfolios◦Student Survey◦Faculty Survey◦Focus Groups

ePortfolio Information

 Electronic portfolios offer three distinct benefits: 1)  They are a creative, lively form in which to show

a student’s best work; 2)  They allow students to gain an awareness of a

larger audience (a portfolio can be published on an intranet or the Internet or saved to a CD and given to college representatives, employers, or anyone a student wants to impress);

3)  They can evolve and grow over a student’s entire academic career.

 "From Worn-Out to Web - Based : Better Student Portfolios." Phi Delta Kappan 85.10 (June 2004): 792-794. Academic Search Premier. EBSCO. [Library name], [City], [State abbreviation]. 27 Aug. 2009 <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=13292501&site=ehost-live>.

ePortfolio Information

The portfolio not only offers a tool for authentic assessment but also a means for students to be reflective practitioners, emphasizing the how and why as much as the what. Time spent in portfolio assessment is not time taken away from teaching or academics, but time refocused and redefined, with the portfolio viewed as a natural complement to learning.

Lombardi, Judy. "To Portfolio or not to Portfolio: Helpful or Hyped?." College Teaching 56.1 (Winter2008 2008): 7-10. Academic Search Premier. EBSCO. [Library name], [City], [State abbreviation]. 27 Aug. 2009 <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=31161341&site=ehost-live>.

ePortfolio Information

Assessment findings derived from students, tutors, and teachers indicate that electronic portfolios have had several positive effects on student learning. These portfolios vividly record writing as a process, providing students and teachers with an effective means of assessing the development of that process over a semester. In addition, they effectively display the final results of the semester, including a student’s self-assessments in the form of written reflections.

Click, Ben A., and Sarah C. Magruder.. "Implementing Electronic Portfolios for Performance Assessment:A Pilot Program Involving a College Writing Center." Assessment Update 16.4 (July 2004): 1-15. Academic Search Premier. EBSCO. [Library name], [City], [State abbreviation]. 27 Aug. 2009 <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=13998617&site=ehost-live>.

Assessment via ePortfolios

Faculty Administrators

Used for student and course assessment

Students upload all formal assignments to portfolios

Allows instructors to look at student progress throughout the course

Students write reflections on various aspects of the course, or on the course as a whole

Used for course and program assessment

Random selection of portfolios to review across courses

10% of portfolios assessed

Look at assignments for writing, critical thinking, and information literacy elements

Sample ePortfolios

www.stevecuello.v2efolioworld.mnscu.edu www.kristycancel.v2efolioworld.mnscu.ed

u

www.luzcampusano.v2efolioworld.mnscu.edu

Course Survey

Used by instructor for course assessmentProvides the instructor with feedback from

his or her studentsThrough narrowly focused questions,

specific feedback is received on:◦Exact assignment◦Support systems in place (i.e. The Writing

Center, eTutoring.org)◦Text (book and reading selections)◦Instructor◦Grading Rubric

Student Survey & Results

1. When you registered for this course, were you aware that it was a Writing Intensive course? 68% Yes

2. What is your overall opinion of the material covered in the course?

3. Would you take another WI course? 93% Yes4. What is your experience with eTutoring.org?5. What is your experience with the Writing

Center?6. What did you like best about this course?7. What, if anything, would you

change/add/remove from the course?

Student Testimonials

“I like to read more because of this class and Professor Bender.”

“I never thought I was going to have so much fun in a Literature course or on an intensive writing course. I’m glad it was a great experience.”

I definitely improved my writing skills. I think if this was just an intro to lit class it would be boring, the writing is what made it fun. I feel so much more confident in my writing.”

WI Student Survey

Used by WI Administrators for program assessment

Conducted across all WI coursesNeither students nor courses identifiedAllows students to report on various

aspects of the WI experience◦Workload◦Support services◦Self-assessment on improvement in writing,

critical thinking and information literacy

WI Student Survey Results

Were you aware that you registered for a WI course? ◦57% Yes◦43% No

Rating of services◦eTutoring

Excellent or Good 68.2% Fair or Poor 20.5% Did not use 11.4%

◦Writing Center Excellent or Good 62.2% Fair or Poor 6% Did not use 31.8%

WI Student Survey Results

Rating of services◦Libguides

Excellent or Good 50.4% Fair or Poor 13.5% Did not use 36.1%

◦ePortfolios Excellent or Good 44.7% Fair or Poor 24.2% Did not use 31.1%

Rating of skills◦Writing

Improved 71.5% About the same 27.7%

WI Student Survey Results

Rating of skills◦Information literacy

Improved 63.4% About the same 31.3%

◦Critical Thinking Improved 66.4% About the same 29.1%

◦Technology Improved 50.7% About the same 42.5%

Faculty Survey

All WI Faculty surveyedSurvey administered online, anonymousAllows faculty to report on services offered in

the Writing Initiative◦Institutes◦Professional development opportunities◦Support services

Faculty report on observations of student improvement in the three areas

Report on own improvement in technology and media use and use of support services

Faculty Survey Results

Familiar with Initiative goals for students◦Use the writing process: 100%◦Support of claims with specific evidence: 89%◦Edit writing according to the rules of standard

academic English: 100%◦Evaluate sources for credibility and academic

appropriateness: 89%◦Employ techniques for integrating information:

67%◦Exhibit the ability to think critically: 100%◦Cite sources using an appropriate

documentation style: 89%

Faculty Survey Results

100% found the goals helpful for improving student learning

67% used rubrics for evaluation before teaching a WI course

87% used rubrics for evaluation in WI courses

33% made use of available tutoring services for their classes before teaching WI courses

Faculty Survey Results

100% incorporated tutoring into their WI courses

67% felt their students were more comfortable doing and integrating research after taking a WI course

78% felt their students were more aware of critical thinking after taking a WI course

89% felt that WI components had value for their other courses

Focus Groups

Conducted by Institutional ResearchRecorded discussion of strengths and

weaknesses of programWI Administrators and WI Faculty met

separatelyIR wrote report analyzing the results for

each group and making recommendations

Focus Group Results

Faculty Administrators

Felt that there were too many technological elements required in the course

Wanted an increase in communication among WI faculty. They felt well-supported by the staff and often communicated with them, however.

Desire more help from the college community as a whole to overcome obstacles to program success

See communication, especially among faculty and program administrators, as key to program success

For the Future

Intro. to Literature Writing Initiative

Work more closely with counselors/registrar/advisors to ensure that students are aware of the course they are taking

Possibly incorporate more use of the Writing Center

Increase the amount of available tech support to allow for easier ePortfolio use

Better marketing of WI courses to students

Better marketing of the Writing Center to students

Better marketing of professional development opportunities to WI faculty

Allow faculty to use either paper or online portfolios for their classes

Thank you!

Kelly Bender: kbender@pccc.eduElizabeth Nesius: enesius@pccc.edu

Portfolio Results

Writing◦Minimum pass rate or higher: 77%◦Below passing: 23%

Critical Thinking◦Minimum pass rate or higher: 25%◦Below passing (or no evidence of CT): 75%

Information Literacy◦Minimum pass rate or higher: 50%◦Below passing in one or more areas: 50%

Sample ePortfolios

Sample ePortfolios

Sample ePortfolios