fshn report spring 2008_radar_version-1
TRANSCRIPT
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
1/18
Food Science Outcomes - 1- May 8, 2008
Food Science Assessment ReportDraft
Spring 2008May 8, 2008
IntroductionThis report of Food Sciences assessment focuses on the Programs use of its Criticaland Integrative Thinking rubric as applied in a set of courses taken by Food Sciencestudents and other students. The Food Science faculty have a variety of goals. Toaddress departmental diversity, meet process strategies for accountability required bythe professional accrediting association (The Institute of Food Technologists), WSU andUI strategic plan and WSUs regional accrediting bodys requirements, and, mostimportantly, to improve the education of future Food Scientists, faculty in the FoodScience programs at WSU and at the UI have identified a single, comprehensiveoutcome: Students who complete the program will demonstrate professional levelcompetence in scientific reasoning in Food Sciences. This is the fifth assessment cyclefor the program.
Goals andOutcomesThis assessment has three goals:
1. The assessment of student performance in Food Sciences overarchingoutcome demonstrate competent scientific reasoning in order to improve theability of students to integrate information from multiple disciplines and to reasonscientifically and critically.
2. The assessment supports WSU and UI institutional goalsquantitative andsymbolic reasoning, critical and creative reasoning, information fluency, studentscommunication skills and their understanding of self and society.
3. The assessment will help identify weaknesses and strengths to help improve andpromote the Food Science program.
Procedures and Methods
To assess scientific reasoning A comprehensive sample of 48 student papers weregathered from various courses at strategic points in the curriculum and rated by 10participating faculty. Student papers from FS 470 (n=13), FS 303 (n=15),FS 220 UI(n=5),FS 220 WSU (n=15) were selected and scored based on six criteria identified by
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
2/18
Food Science Outcomes - 2- May 8, 2008
Food Science faculty and an assessment consultants from CTLT to comprise thescientific reasoning construct.
Faculty agreed that the double review process used by the Educational TestingServices would be appropriate for establishing and monitoring inter-rater reliability and
consensus necessary for improving program coherency and faculty community. Toestablish reliability, raters participated in a norming session, rating an identical paperbased on the scoring criteria, and then negotiated a consensus on how the criteriawould be applied. Further, by adopting the rigor embodied in the double reviewapproach faculty establish expert validityin the assessment process.
In the rating process, participants agreed that competent performance would bereflected in a score of 4 or better on the 6-point rubric. Each paper was assessed by tworaters on each criterion of the rubric using a scale that ranges from 1-6. A score of 4designates the desired level of competence of a WSU graduate; 6 represents
exceptional or mastery level performance. A competent student paper thereforerequires consensus of two independent raters. Scores that differed by more than onepoint were reconciled by a third rater and the papers were further tagged for additionalreview.
Performance Criteria
In order to effectively demonstrate competent scientific reasoning in FS, studentsshould be able to demonstrate the six essential subordinate or supporting outcomesthat are aligned with WSUs institutionally generated student outcome goals and the UI
5 program learning outcomes. Students who complete the FS curriculum will thereforedemonstrate the ability to:
1. Identify, summarize, and define the issue or problem at hand.2. Clearly state purposes, objectives, or hypotheses.3. Present observations and results in a complete, logical and clear fashion.4. Assess, discuss, and reconcile the supporting data/evidence acquired from the
exercise/experiment in relation to the existing scientific literature.5. Identify and assess conclusions, implications, and consequences.6. Effectively organize and articulate information to promote understanding and
communicate significance of the issue or problem.
The six criteria can be mapped to the WSUs Six Goals of the Baccalaureate byassociation as illustrated in Figure 1 and Figure 2:
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
3/18
Food Science Outcomes - 3- May 8, 2008
Figure 1. Critical and Integrative Thinking in Food Science
Figure 2 illustrates how the 6 institutional outcomescritical thinking, quantitative andsymbolic reasoning, information literacy, communication, specialty, and self in societyare embedded in the Food Science assessment strategy.
Figure 2. Mapping to the WSUs Six Goals of the Baccalaureate
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
4/18
Food Science Outcomes - 4- May 8, 2008
FS Outcomes Mapped to the WSU Six Goals of the Baccalaureate
The outcomes assessed in FS implicitly map to WSUs six goals of the baccalaureate.The correspondence between WSUs six goals and the FS assessment process hasbeen used here to provide a view of how FS students are performing on institutional
goals. Table 1 reflects a distribution of rubric dimensions to the WSU Six. Figure 3represents the results of that distribution.
FS Outcomes Allocation to WSU6Critical
ThinkingQuant &Symbolic
Reasoning
InformationFluency
Communication Self &Society
Specialty
Problem 3.49 3.49 3.49 3.49
Hypothesis 3.34 3.34
Observation 3.38 3.38 3.38 3.38 3.38
Data 3.23 3.23 3.23
Conclusions 3.18 3.18 3.18 3.18 3.18
Communication 3.40 3.40
WSU 6 3.32 3.28 3.32 3.40 3.35 3.34
Table 1. A distribution of rubric dimensions to the WSU Six
Figure 3. A Radio Graph of the Distribution of Outcomes to the WSU 6
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
5/18
Food Science Outcomes - 5- May 8, 2008
Results for 2008
Inter-rater Reliability
Inter-rater reliability was determined using the ETS methoda calculation of the
percentage of overall agreementwhich was used to monitor the reliability of the raters.Acceptable levels of reliability are indicated by 75% or better using the ETS method.The overall percentage agreement among raters in this session was 74%.
Rubric Dimension Course Student Papers Inter-Item Reliability
FS 470 FS 303 FS 220 UI
(n=13) (n=15) (n=20)
ETS ETS ETSProblem 85% 73% 75% 78%Hypothesis 77% 80% 70% 76%Observations 87% 73% 65% 75%Data 100% 93% 90% 94%Conclusions 69% 67% 65% 67%Communication 62% 60% 85% 69%
Average 80% 74% 75% 76%
Table 2. 2008 Inter-rater Reliability by Course and Dimension
Figure 4. Inter-rater reliability by dimension
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
6/18
Food Science Outcomes - 6- May 8, 2008
Student Performance
Student performance is reported as an average paper rating by class and overall.Future reports might include percentage of papers rated as competent at the level of 4or better.
Course
FS 470 FS 303 FS 220WSU and UI
Overall
Rubric Dimension(n=13) (n=15) (n=20) (n=48)
Mean Mean Mean Mean
Problem 3.81 3.10 3.57 3.49Hypothesis 3.65 3.27 3.10 3.34Observations 4.04 2.97 3.12 3.38Data 4.00 2.73 2.97 3.23
Conclusions 3.77 2.77 3.00 3.18Communication 4.00 2.97 3.23 3.40
Total Average 3.88 2.97 3.17 3.34
Table 3. 2008 Critical and Integrative Thinking Outcomes by Dimension and Course
Figure 5. Overall Ratings by Class
Assignment Assessment
In addition to assessing student performance in written papers, raters used the samecriteria to assess the assignments that students were responding to. The scoringmethod assessed the extent to which the assignment expectations related to the sixdimensions of the rubric: explicit, implicit, or absent. These ratings are necessarily
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
7/18
Food Science Outcomes - 7- May 8, 2008
approximate, as they do not reflect the spoken instructions / discussions that proceededor accompanied the assignment or feedback students may have received as theyworked or revised (with or without a rubric). Nonetheless, the results are reported hereto inform faculty reflection on the alignment of the assignments with program goals.
Figure 6. Comparison of FS 303 Assignment Ratings
Faculty Feedback on the FS 303 Assignment Prompt
Dimension 1: Problem
Explicit in works and phases but no explanation given (R1)
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
8/18
Food Science Outcomes - 8- May 8, 2008
Hard to tell if student is to write a 'term paper' or 'proposal' these are different things.
Assuming that the topic is pretty open ended (but has to have a theoretical and
applied component), and this was probably presented in class, but is not clear from
the writing here. (R3)
Does not clearly explain the need to identify or embedded or implicit aspects. (R4)
Not entirely clear what you want. No introduction to the assignment given. I could
not do this assignment without instructor explaining verbally. (R5)
Dimension 2: Hypothesis
Explicit in works and phases but no explanation given (R2)
This is pretty clear from the outline, but presumes that a student knows what this
means already. I presume that there was explanation of this in class. (R3)
Purpose or hypotheses not requested explicitly. (R 5)
Dimension 3: Observations
Explicit in works and phases but no explanation given (R2)
Noted in outline but no explanation provided (R3)
More detail of expectations needed.(R5)
Dimension 4: Data
A little hard to figure out how to do this in light of the first sentence (R3)
More detail of expectations is needed. (R5)
Dimension 5: Conclusions
Explicit in works and phases but no explanation given (R3)
hard to figure out what the student is supposed to be concluding (results of
earlier studies) or push ideas and convince reader that this work would be worth
funding. (R3)
Does not address consequences (R4)
More detail of expectations needed. (R5)
Dimension 6: Communication I would have trouble trying to organize my thoughts for this, the structure is clear,
but would be difficult to implement. (R3)
This really lacks detail on expectations. I could not do this assignment without
instructor explaining verbally. (R5)
Assignment does not express the requirement for:
o Title is sufficiently descriptive and informative.
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
9/18
Food Science Outcomes - 9- May 8, 2008
o Text is clear and well organized.
o Text contains few or no spelling or grammatical errors.
o The overall style and organization consider the audience, follow a logical
order, and unify all aspects of the project to maximize understanding of
the problem or issue.
o Supporting literature is fully and accurately cited (R4)
Figure 7. Comparison of FS 220 Assignment Rating
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
10/18
Food Science Outcomes - 10- May 8, 2008
Faculty Feedback on the FS 220 Assignment Prompt
Dimension 1: Problem
Written English is a bit difficult to follow and some instructions are not all that clear
(R3)
Dimension 2: Hypothesis
Critique does not usually require that there be a hypothesis (for sure). An outline
should include objectives or a roadmap for the paper, but whether this would be
understood by students would depend on what level they are at in college.(R3)
Dimension 3: Observations
Could explain more (R5)
Critique of a current topic does not necessarily involve analysis of any type ofdata. If I had this assignment, I would be a bit confused as to what type of data Iwould need to be evaluating. It is possible that the student would not know howthe data were collected - since their primary source documents could be reviewarticles and not reports in the peer reviewed literature. (R3)
Dimension 4: Data
o This implies that the paper will involve an analysis of primary literature and notreview literature and also that the paper will focus on scientific reports ratherthan policy studies.(R3)
Dimension 5: Conclusions
Explain more here (R5) Charts and tables would not belong in this section, but instead in the results. This
should be a written conclusion. (R3)
Dimension 6: CommunicationMore detail here would help. Depending on a students perspective of what this
assignment is supposed to be about (e.g. science vs policy)they could have
major problems with trying to write a paper with the proposed structure. The
assignment is "to critically analyze information about a topic related to foodsafety and quality and to write a paper which critiques, summarizes and
articulates the current level of knowledge concerning that topic". This is pretty
vague. (R3)
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
11/18
Food Science Outcomes - 11- May 8, 2008
Figure 8. Comparison of FS 470 Assignment Rating
Faculty Feedback on the FS 470 Assignment Prompt
Dimension 1: Problem
Pretty clear (R3)
Dimension 2: Hypothesis
Objectives are clear but request for hypothesis is not (R3)
Dimension 4: Data
No mention of the use of statistics (R4)
Dimension 6: Communication
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
12/18
Food Science Outcomes - 12- May 8, 2008
Assignment does not fully express the requirement for:
o Title is sufficiently descriptive and informative.
o Text is clear and well organized.
o Text contains few or no spelling or grammatical errors.
o The overall style and organization consider the audience, follow a logical
order, and unify all aspects of the project to maximize understanding of
the problem or issue.
o Supporting literature is fully and accurately cited (R4)
Good assignment to fit the rubric. (R5)
Comparison of Student Work and Assignment Prompt Ratings
Figure 9. Comparison of Ratings for Student Work and the Assignment Prompt for FS202
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
13/18
Food Science Outcomes - 13- May 8, 2008
Figure 10. Comparison of Ratings for Student Work and the Assignment Prompt for FS303
Figure 11. Comparison of Ratings for Student Work and the Assignment Prompt for FS470
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
14/18
Food Science Outcomes - 14- May 8, 2008
Discussion and Next Steps for 2008 2009
Make the Most Out of Results1. The next step will be to meet to discuss what the assessment data means and to
identify and implement program changes. This meeting will solicit facultyreflections on strategies for revising assignments to improve program outcomes.(The reflections are invaluable evidence of quality enhancement essential foraccreditation as well as program improvement.)
The program has made solid progress in formalizing the assessment of the threegoalsstudent competence in scientific reasoning, relating the assessment of thatcompetence to the WSU Big Six, and implementing a process that will help establishFood Science.
Program Innovations
Throughout the study, noteworthy innovations have been integrated into Food Science:
1. A formalized, annual time-line for program assessment management wasdeveloped. This model is a one year cycle oriented toward student assessment inselected classes.
2. A schedule of ad hoc faculty meetings with participating faculty to discuss
assessment issues and strategies for revising assignments and prompts to improveprogram outcomes was implemented.
3. Changes were incorporated in Food Science syllabi to expand the scientificreasoning concept and to incorporate it as a key aspect to the assignments.
4. The assessment of student work and assignment prompts was completed onlineusing the Skylight survey tool. This innovation offers the option of more convenientasynchronous rating after a brief face to face norming session.
Selecting Next Steps
We make suggestions below for next steps but without investment from faculty progressis unlikely. Ultimately the Food Science faculty will have to make a commitment toimprovements of their choice, identify the players needed to implement them, andcreate a timeline with milestones. Such choices, even if limited in number, will guidefuture assessment efforts in the classroom settings experienced by Food Sciencestudents.
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
15/18
Food Science Outcomes - 15- May 8, 2008
Suggestions for next steps
Measure what you value and value what you measure. If the learning outcomesidentified by faculty are important enough to guide the assessment of the program, they
are important enough to communicate to students as a guide for their learning.
There are many ways to communicate the value of the learning outcomes to thestudents.
Give the rubric to students when they enter the program. Make it clear to them thatthese criteria will be the measure of their success in the program and of theirreadiness to enter the professional community. . The grading scale is absoluteprofessional competencefaculty, professionals in the industry, peers, and theywill continuously assess their performance, using that absolute standard, as theymove through the program
Let students know that through ongoing and consistent feedback they willimprove in their understanding and practice of these criteria. Coach the studentsin using the rubric to self assess and assess peers. (CTLT can help with this)
Involve Students in the Process: On the first day of classes, give a copy of therubric to your students along with the syllabus.
o Conduct a short norming session with students early in the semester.
o Active/Collaborative Learning: Incorporate assignments into the coursethat require students to give each other feedback on their use /interpretation of the rubric.
o Encourage Revisions: Allow students to revise written assignments inconsultation with the rubric and have them turn in both first and reviseddrafts with indications of where the rubric was used to revise.
Coach faculty in using the rubric to self assess, assess peers, and students.(CTLT can consult with faculty individually or in group workshops)
Alignment: Monitor thealignment of assignment prompts with the outcomescriteria as identified in the rubric.
Invite students, TAs, professionals, community colleagues, andadministration to participate in rating sessions. Involve them in assessment ofstudent work, assessment and revision of assignments, and discussion ofinnovations.
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
16/18
Food Science Outcomes - 16- May 8, 2008
Make Use of Rating Session Results
Meet regularly to identify and implement changes. Identify trends and problemareas in student performance. Solicit faculty, professional and student reflectionson strategies for revising assignments to improve program outcomes (The
reflections are invaluable evidence of quality enhancement essential foraccreditation as well as program improvement).
Develop a formalized timeline for rating sessions and the use of their results.The model below repeats as part of an ongoing process
Model Time-line for one Semester in an Assessment CyclePre-term Mid-semester Post-term
Design course,assignments andactivities.
Collaborativelyreview thesematerials toidentify andenhance thealignment ofembeddedoutcomes criteria.
Provideoutcomes criteriato students andopportunity tonorm on samplework.
Use criteria tosupport studentlearning (facultyfeedback, peerfeedback, self-reflection).
Provideopportunities forstudents torevise their workbased onfeedback.
Provide explicitmid-termfeedback tostudents usingcriteria andcollect feedbackfrom studentsabout criteria andassignments.
Examine how thecriteria are beingused; facultyshareobservations andmakeadjustments torubric,assignments,processes, etc.
Continue usingcriteria tosupport studentlearning.
Conduct outcomesassessment of studentwork and of assignments.
Use results to enhancealignment of outcomes ,assignments, activities,and to identifyrefinements for nextcycle.
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
17/18
Food Science Outcomes - 17- May 8, 2008
2008Food Science, with assistance from the Center for Teaching, Learning, & Technology
WWWaaassshhhiiinnngggtttooonnn SSStttaaattteee UUUnnniiivvveeerrrsssiiitttyyy
-
8/14/2019 FSHN Report Spring 2008_radar_version-1
18/18