2015 mcas-alt: how to submit a “complete” portfolio karen orlando mcas-alt teacher consultant

2015 MCAS-Alt:

How to Submit a “Complete” Portfolio

Karen OrlandoMCAS-Alt Teacher Consultant

Topics

Top five reasons portfolios were scored Incomplete

Other reasons for an Incomplete score

Checking for “completeness”

Frequently-asked questions 2

Massachusetts Department of Elementary and Secondary Education

Top Five Reasons for Incomplete Portfolios

3


Top Five Reasons For Incomplete Portfolios

1. Some or all brief description(s) did not address the measurable outcome. As a result, student did not address measurable outcome on at least 8 dates.

2. Brief descriptions were not included or did not provide enough information on at least 8 dates.

3. Titles and/or samples of text used with student were missing in the ELAReading strand.

4. At least two pieces of primary evidence were not submitted that related to measurable outcome listed on the data chart.

5. No evidence of self-evaluation in the strand. 4



1. Some or all brief description(s) did not address the measurable outcome. As a result, student did not address the measurable outcome on at least 8 dates.

5



6

Walter will identify basic forms of energy with 75% accuracy and 75% independence.


7

Student will answer simple comprehension questions to demonstrate the understanding of an informational text with 80% accuracy and 80% independence.

Brief Descriptions Unmatchedto Measurable Outcome

Page 35: “…include a brief description beneath each data point that clearly describes what the student did and how the student addressed the skill, taking particular care to document that the activity matched the measurable outcome .”

Page 35: “…assess and record data only on those skills directly based on the measurable outcome. The data will be inconclusive, and the chances of scoring Incomplete will increase, when unrelated or multiple skills are included on the same data chart.

Page 35 2015 Educator’s Manual for MCAS-Alt

8


Creating a Measurable Outcome from Entry Points that Include Related SkillsExample:Entry Point as written:

“Solve number sentences that represent one-step multiplication and division word problems.”

Option 1: Create a measurable outcome that addresses both skills. All activities must document “solving number sentences

involving one-step multiplication AND division problems.”

Option 2: Modify the measurable outcome: “Solve number

sentences that represent one-step multiplication word problems.”

All activities must document “solving number sentences involving one-step multiplication problems.”

9



10

Which descriptions are scorable?

Measurable Outcome: Student will solve addition and subtraction word problems with 80% accuracy and 100% independence.

Student solved addition and subtraction word problems worksheet.

Student solved subtraction word problems in math class.

Word problems game with whole class

Solved addition and subtraction word problems created by peers on smart board

Solved addition problems on computer with no help


11

Clear and Specific Descriptions Score accuracy and independence only on the skill related

to the measurable outcome. Clearly indicate this on the work description label.

Example: Skill measured in the measurable outcome:

“Answer questions about bar graphs”

Activity: Student creates a bar graph from the results of a class survey, then answers questions about the graph.

Scorable Description: “After creating a bar graph from a class survey, student answered questions about the graph. (Accuracy and independence were based on questions only.)”

Unscorable Description: “Student created a bar graph and answered questions.”


2. Brief descriptions were not included or did not provide enough information on at least 8 dates.

12



13

Walter will compare fractions using the terms “greater than,” “less than, or “equal to” with 80% accuracy and 80% independence.

Scorable Brief Descriptions

Page 35: “include a brief description beneath each data point that clearly describes what the student did and how the student addressed the skill, taking particular care to document that the activity matched the measurable outcome;

14


Pages 35 and 40 2015 Educator’s Manual for MCAS-Alt


15

Scorable Brief Descriptions continued Page 40: “Brief descriptions are provided

below each learning activity that match the measurable outcome, and describe what the student was asked to do to address the measurable outcome, and how they did it. The following are examples of appropriate brief descriptions:…

Pages 35 and 40 2015 Educator’s Manual for MCAS-Alt


Acceptable Brief DescriptionsMeasurable Outcome for STE- Technology/Engineering Walter will identify simple machines with 80% accuracy and 100% independence.

Date (m/d/y)

9/13/12 10/15/12

10/17/12

10/24/12

11/4/12

11/15/12

11/20/12

12/7/12

What the student did (skill)

How did they do it? (approach materials)

Identified simple machines by labeling pictures in an adapted textbook

After listening to “Simple Machines,”identified simple machines in the book by pointing

Homework:Identified simple machines at home, made a list of the ones he found

Worksheetidentified inclined planes and levers as simple machines using pictures and text

After watching video, Wheels and Axles,identified wheels and axles around the school

Using Home Depot flyer, Walter identified levers and wedges using a bingo dauber

After watching Eduhead on the computer, Walter identified inclined planes by matching them to the pictures in the video

Class created a poster of simple machines he identified the simple machines in of each picture with a label

16

These brief descriptions DO NOT provide enough information to know what the student was asked to do, or how he/she did it.

BriefDescription

Pre-test

Solve word problems on worksheet

Home-work

Digestive system class assign-ment

Answer vocab questions

Label parts of digestive system

Describe digestive system

Chap. 2 test

17



18

Which descriptions are scorable?

Student answered questions about a story during speech class

Student answered questions on an informational text

After listening to Boston Globe article on “Pollution,” student answered simple comp. ques.

After listening to chapter 1 of Helen Keller by M. Davidson, student answered simple comp. ques.

Comprehension question quiz, chapter 2

Outcome: After listening to story or article, student will answer simple comprehension questions to demonstrate understanding of an informational text with 80% accuracy and 80% independence.


3. Titles and/or samples of text used with student were missing in the ELAReading strand

19



20

Titles of text are missing!


21

Cite the Title or Include Actual Text Page 15-19: “The title or a photocopy of the text

used for assessment must be included in all brief descriptions for ELAReading; a copy of any teacher-created text must be submitted”

Page 41: “Evidence or brief descriptions in the ELAReading strand must refer by name to the text from which words, phrases, or excerpts were selected for assessment, and must include either: the title of the published text, or

the actual text (e.g., sentence, passage, narrative, etc.), or

a photocopy or printout, if text is teacher-created, untitled, or includes selections from a print or digital source (e.g., websites)

Pages 15-19 and 412015 Educator’s Manual for MCAS-Alt


22

Options for Citing the Text Provide titles. Provide copy of teacher-created text,

worksheets, or selections from websites. Attach a list of text used to data chart with

dates and names of text used. Include topic of the text if it is from a specific

reading program, rather than just name of program.

Include title of text and/or topic if it is from a specific website, rather than just the name of the website.

For untitled texts include a photocopy


4. At least two pieces of primary evidence were not submitted that relate to the measurable outcome listed on the Strand Cover Sheet

23



24

Student will compare and contrast characters in a story with 90%accuracy and 80%Independence

25


EVIDENCE #1

Student


26

EVIDENCE #2

Student

Student is answering questions

about characters,

not comparing

and contrasting

them.

Pages 12-19 “…two additional pieces of primary evidence showing the student’s performance of the same targeted skill identified on the data chart.”

Page 33 “A minimum of one data chart and two pieces of additional primary evidence (called the “core set of evidence”)…”

27


Pages 12-19 and 33 2015 Educator’s Manual for MCAS-Alt

Two Pieces of Primary Evidence


5. No evidence of self-evaluation was found in this strand.

28



29

Self-Evaluation

Either… Evidence was either missing altogether,

or Showed no evidence of student choice

Stickers Stamp “Student smiled” “Student traded in for music”

Self-Evaluation

“Self-evaluation activities document the student’s choices, decisions, and involvement before, during, and after instruction, including evidence that the student performed…”

30


Pages 39 2015 Educator’s Manual for MCAS-Alt

Examples of Self-Evaluation include: planning and goal setting choosing an activity or next steps in an activity selecting a problem-solving strategy monitoring own progress or use of a strategy (e.g., by

checking off steps as each is completed) deciding when to continue or end participation in an activity identifying and correcting his/her own errors graphing own performance or progress on a chart determining own score using a rubric selecting work for his/her own portfolio reflecting on his or her performance; for example, teacher

asks student to respond to: What did I do well? What am I good at? Was this too

easy? What did we do during this activity? What did I learn? How could I do better? Where do I need help? What should I work on next? What would I like to learn?

31



32

Other Reasons for Incomplete Literature and Informational text were combined in a

single strand. The Reading strand must be one or the other.

Data chart did not document student performance on at least 8 different dates.

Both Accuracy and Independence began above 80-100%.

No data chart was submitted. Data were unclear or could not be interpreted by the

scorer. Percent accuracy and/or independence were not

included on primary evidence and/or could not be determined by scorer.

Student's name and/or date (month, day, year) was missing on one or more pieces of primary evidence, or was incorrect on Work Sample Description label.

33



34

Level of Complexity = 1

What Does Level of Complexity = 1 Mean?

Strand must assess standards required for a student in that grade.

Data chart must document student performance of an acceptable measurable outcome in the required strand/domain. (Teachers cannot make up or use outdated entry points!)

ELA Reading is comprehension of words, phrases, sentences, passages based on a text.

Access skill must be taught in the context of an academic activity.

35


Portfolio strand reflects little or no basis in,

or is unmatched to, Curriculum Framework learning standards required for

assessment.

Page 48 Educator’s Manual for MCAS-Alt

Skill Not Taught in the Context of an Academic Activity

36


Earth and Space Science is not assessed in High School

37


Checking for Completeness

38


12 Key Questions to Check for Completeness

1. Does the strand include a data chart and at least two pieces of primary evidence

2. Is the first data point on the data chart below 80 percent accuracy and/or independence?

3. Does the data chart have at least 8 different valid dates?

4. Is the student’s name, date, % accuracy, and % independence listed on the data chart and on at least two pieces of primary evidence (or on a Work Description label)?

39


Pages 35-36, Educator’s Manual for MCAS-Alt

12 Key Questions to Check for Completeness (continued)5. Do brief descriptions on the data chart state what the student was asked to do (i.e., what was the activity?) and how the activity was conducted (instructional approach, materials, etc.) on at least 8 different dates ?

6. Do brief descriptions on at least 8 different dates address the skill identified in the measurable outcome?

7. Do at least two pieces of primary evidence directly address the measurable outcome?

40



41

12 Key Questions to Check for Completeness (continued)8. Are all skills listed in the measurable outcome (e.g., “addition and subtraction”) addressed in at least 8 activities on the data chart and two pieces of evidence (all skills must be addressed in each activity).

9. If photographs or video are included as primary evidence, do they show a final product and are they clearly labeled?

10. If a teacher-scribed work sample is included as primary evidence, does it provide sufficient information to determine what the student did and how the student did it?


42

12 Key Questions to Check for Completeness

11. Do brief descriptions include titles of all texts?

12. Do all activities in the strand include either Informational or Literary text (but not both)?

For ELA-Reading Strands Only:

Common Questions

43


How Do I Determine Accuracy and Independence?

First, determine what the outcome is. What are you asking the student to do?

Then, determine the activity the student will perform to address that outcome.

Next, divide that activity into “items” or steps. Determine the overall accuracy (correctness)

and independence (i.e., no prompts needed) for the entire activity:

% Independence = Number of “items” performed independently

Number of possible responses

44


Examples of “Items”:Outcome: Andre will answer simple comprehension questions in relation to a text with 80% accuracy and 100% independence.

Activity: After reading Chapter 1 in “Frog and Toad Are Friends” by A. Lobel, with support of visuals and a reading checklist, student answered 4 comprehension questions on a worksheet.

45


Page 38 2015 Educator’s Manual for MCAS-Alt

Question#1 Correct (accurate) Visual prompt(not independent)

Question#2 Correct(accurate) No Prompt (independent)

Question#3 Correct (accurate) Gestural prompt (not independent)

Question#4 Incorrect (inaccurate)

No prompt (independent)

Averages for all activities

3/4 or 75% accurate

2/4 or 50% independent


46


47

• How many questions? 5

P

I

I

I

I

• How many correct responses? 2• What is the accuracy score? 2/5 or 40%

• How many independent responses? 4

• What is the independence score? 4/5 or 80%

Independence: Prompts vs. Accommodations

The following are cues and prompts since they guide the student toward the answer:

1. Verbal assistance (coach or tell: ”look on page…”)2. Gesture (point or indicate where to find answer)3. Visual cue (provide a sample problem or model) 4. Full or partial physical assistance (including

hand-over-hand assistance)

Any Cue or Prompt = a “non-independent response”

48


Pages 37-39, 50Educator’s Manual for MCAS-Alt

What is Generalized Performance?2015 Educator’s Manual: Page 39: “Generalized performance reflects the

student’s ability to transfer knowledge and/or skills to other learning situations through application of a range of instructional approaches and activity formats, including any of the following instructional elements:· media and materials (e.g., using art materials, written

text, manipulatives, computer)

· activity formats (e.g., classroom projects, small group discussions, paired research, experiments)

· presentation formats (e.g., oral, written, multimedia)

· method of response (e.g., handwritten, word-processed, oral, creation of a visual display, video)”

49


Page 39,2015 Educator’s Manual for MCAS-Alt

What is a Teacher-Scribed Work Sample and When Should it Be Used? One method of documenting performance of

a student who does not produce written work

Indicates what the activity was, what response options were offered to student, and how he/she responded on each item (accurately, independently?)

Documents student’s performance of a series of tasks recorded by the teacher at one time. 50


Page 37,2015 Educator’s Manual for MCAS-Alt

Teacher-Scribed Work Sample

· What was the activity?

· What materials were used?

· What was the correct response?

· What was the student’s actual response?


Activity

Materials

Correct Respons

e

Student’s

Response

51

Teacher-Scribed Work Sample: Not Enough Information to Score

• What was the activity?

• What was the correct response?

• What was the student’s actual response?

• What materials were used?

52


Reminders Double-check the evidence before submitting

• Have another teacher or supervisor double-check your evidence and see if they can answer “yes” to the 12 questions.

Check for clerical errors such as incorrect dates, names or work of another student, or evidence submitted in wrong strand.

Attend Portfolios-in-Progress review sessions (January and March) and have training specialist look over your portfolios, answer questions, and share ideas

Make sure you are using current Educator’s Manual, Resource Guides, and forms

53


2015 mcas-alt: how to submit a “complete” portfolio karen orlando mcas-alt teacher consultant

Documents

measurable outcome page

secondary education

incomplete portfolios

reasons portfolios

number sentences

brief descriptions unmatched

division problems

incomplete score