Response to Peer Review and Instructor Feedback

Julia Webster

Azusa Pacific University

Peer Review of the Writing Section of My Achievement Test

Name of Test: Unit Six Test Date: 3-8-2016

Developer's name: Julia Webster Reviewer's name: Paul Wendler

1. Test Purpose: What do you understand to be the purpose of this test? What specific competencies are being addressed?

I understand that this test is assessment of content taught in unit six of Julia’s class. Julia writes, “As far as writing skills, the curriculum for the unit covered U.S. holidays, a bad day at work, writing an absence note, and past tense grammar”. Writing competencies are being addressed.

2. Test Design: How would you classify the techniques used in this test?

a. Picture descriptions technique

b. Word order technique

c. Letter writing technique

3. Test Appeal: What do you find appealing as a test taker? What do you find frustrating?

I find it appealing that this is only a little amount of text that I need to read in order to take the test. This makes it valid to me because I can see what I am being tested on. Not a reading test but a writing test. Also I like the spacing and look of the second part that tests word order. Very clear and well organized.

Frustrating is the small amount of space for writing my description of the pictures and letter. Equally frustrating is the ambiguity of the rubric on which I am being scored.

4. Layout: What do you find attractive or helpful about the layout of the test? How could it be improved?

Pictures are very appealing and easy to understand. Very good job creating borders so that Ss know what is expected and where. The test is easy to understand and directions are clear.

5. Language: How would you evaluate the language used throughout the test in terms of natural usage, grammatical correctness, and mechanics (spelling and punctuation)? Can you suggest any corrections or improvements?

(see comments for more specifics) In general the language used within the test was grammatically correct and showed good mechanics. The first instructions were awkward and I have given some suggestions in my comments on the test. I didn’t see any spelling errors on the test.

6. Directions: Are clear, simple directions provided for each section? Are the expectations transparent in terms of grading criteria? Are any items open to misinterpretation?

Very simple and clear instructions on the test. I like how this test is just about writing and doesn’t try to do much else.

The only thing that could be misinterpreted is that you are asking them to write in the past tense in section A but never say that. As a native speaker I see that you write, “Joe had a bad day yesterday” but I am not sure if ELLs would pick up on that. You may want to explicitly say you want them to write in the past tense.

7. Multiple choice items: Are any of the distractors weak or misleading? Do you find any of the features that should be avoided?

N/A

8. Scoring guidelines: For objectively scored items, do you agree with the scoring key? Can you suggest other equally correct responses? For subjectively scored items, or items for which partial credit is possible, are scoring guidelines and rubrics clear and consistent with the purpose of the test?

The objectively scored items in section B are well done. I am assuming based on the information provided that each question is worth one point. However, I am wondering what happens if just one phrase is out of place? Does the student miss the entire question for one out of place phrase? Also, as suggested in my comments, I think that there should be more questions in section b. This would help it to become more reliable by putting less weight on each question especially if they are scored either right (1 point) or wrong (0).

The subjectively scored items need more consistency and clarity (see notes). The rubric is not complete enough and needs a broader range of criteria. For example, from the looks of the rubric students will either score 0, 2, 5 or 8 and nothing in between those scores. Also the language in each point section needs to be more consistent with the other point sections to ensure reliability.

9. Relevance: Is the test appropriate to the students and purpose for which it is designed? Try to identify specific ways in which the test is well matched or possibly mismatched to the students or curriculum objectives.

Yes, I would say that this test is appropriate as it comes straight from the information being covered in unit six. The test addresses the past tense in both sections a and c as both the letter and paragraph are to be written in the past tense. Holidays are covered in section b although I don’t know if that section is covering the holidays are covering sentence structures or both. Could you cover holidays with an activity that has them writing about the holidays?

10. Validity: Considering both the content and the scoring procedure, try to identify at least one feature of this test that fortifies its validity and one that weakens it.

I believe that the scoring procedure weakens the tests consequential validity because it is too abstract and not consistent enough. Without clearer scoring guidelines the benefit to the students and teachers will not be clear. To avoid this Julie should not only revise the rubric but also find benchmark papers to help her know what she is looking for in order to give each point value.

I believe that the content of the test fortifies it’s validity. All of the items being tested should be familiar to the students. When they see them again on the test it reinforces their importance.

11. Reliability: Similarly, identify aspects of the assessment that might strengthen of threaten the consistency of the results. [Again, try to identify at least one of each.]

Again the rubric’s lack of consistent wording and order could hurt reliability. If the qualifications to receive certain scores are not clear this will threaten inter-rater reliability. \

Also the all or nothing style of part b might have an impact on reliability. If a student knows the correct word order but slips and makes one mistake they get the same score as one who doesn’t write anything.

The borders providing space to answer questions help with reliability. Students know that they are expected to fill in the lines for the description and the box for the letter. This will help the students to produce writing similar in length and give the teacher similar amounts of writing to evaluate.

12. Ethics: Are you aware of any ethical issues that this test might raise?

I do not see any ethical issues.

Unit 6 Test Name:________________________

Date: ________________________

A. Look at the pictures. Joe had a bad day yesterday. Write three or four sentences telling the story of Joe’s bad day.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

B. The following sentences about U. S. holidays have been scrambled. Put the phrases in order to make correct sentences.

Example:are ten / United States / holidays / there / national / in the

Answer: There are ten national holidays in the United States.___________________

in February / celebrate / third Monday / we / on the / President’s Day 1. __________________________________________________________________________

on national / are closed / offices / holidays / government

2. __________________________________________________________________________

July 4 / United States / the birthday / of the / celebrate / we / on

3. _________________________________________________________________________

C. Your child, Sophie, missed school yesterday because she was ill with a high fever. Write a note to her teacher, Mr. Ivan King, to explain her absence.

Scoring rubric for A and C.

Score Criteria

0 - No effort Response was left blank

2 - Fails to meet expectations

Severe errors in grammar, word order and form.Meaning obscured or incomprehensibleSerious errors in sentence structureDemonstrates no mastery of conventions

5 - Meets expectations Occasional errors in grammar, word order and form that do not obstruct meaning.Minor errors in sentence structure.Demonstrates awareness of conventions, with occasional errors.Appropriate word choice, but limited vocabulary.

8 - Exceeds expectations Few if any noticeable errors in grammar, word order and form.Demonstrates mastery of syntactical structure.Evidence of broad and appropriate vocabularyDemonstrates mastery of conventions.

Scoring for Section B

One point for each correctly assembled sentence that is grammatically correct.

Points Possible??

Instructor Feedback

Additional instructor's comments about your submission

Thank you for your explanation, Julia. See more comments below.

Christina

---------------------------------------

Quality of test items, especially in terms of validity and reliability 

Directions are generally clear.

Part A specifies target verb tense. 

Part B objective unclear, but interesting prompt.

Part C has best construction but is least clear in terms of test objectives.

Reliability is fairly strong, esp Part C. But without clear test objectives (targeted language you want Ss

to produce and that you will evaluate), validity will suffer.

Appropriateness of the test for the class described in the Student and Curriculum Profile

Test appropriate for S&C demographics.

Clarity and appropriateness of the scoring guidelines

Rubrics have clear criteria descriptions.

"Exceeds prompt requirements" doesn't seem a fair criteria for a full score. A full score should be

completion of prompt requirements. "Exceeding" prompt requirements seems to move into the realm of

extra credit. Also seems unfair if directions didn't ask Ss to "exceed" requirements to get a full score.

Good inclusion of a range of scores to help scorer have flexibility in determining score to assign based

on how well S met the criteria in the relevant criteria block.

If S chooses to describe Sophie's present condition ("she still has a fever") in Part B, will S's score be

penalized?

Variety of techniques used - Used three techniques

Creativity - Nice illustrations

An outstanding sub-test will have a unifying theme, a coherent flow, and no grammatical or mechanical errors.

Grammar/mechanical error - None noted.

Coherent flow - No obvious flow (maybe chronological?). Tasks should flow from more controlled to

less controlled, so Part C should be first.

Unifying theme - Jose and his daughter 

Writing Section of the Final Unit Test

with Changes in Response to Feedback

18. How many national holidays celebrate specific people? __________________

19. Name one national holiday that celebrates the U.S. military. __________________

Writing

E. José helps his daughter with her homework about U.S. holidays. The following sentences are about U.S. holidays. They are scrambled. Put the phrases in order to make correct sentences. Use a capital letter to begin the sentence. End the sentence with a period. More than one correct answer may be possible. (8 points)

Example:are ten / United States / holidays / there / national / in the

Answer: There are ten national holidays in the United States.___________________

in February / celebrate / third Monday / Americans / on the / President’s Day 20. __________________________________________________________________________

on national / are closed / offices / holidays / government

21. __________________________________________________________________________

July 4 / United States / the birthday / of the / celebrate / we / on

22. _________________________________________________________________________

U.S. Military / all people / honor / we / in the / on Veteran’s Day

23. __________________________________________________________________________

F. José had a bad day yesterday. Look at the pictures. Write three or four sentences that tell the story of his bad day. Use past tense verbs. (8 points)

_____________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

_

G. Sophie missed school yesterday because she was ill with a high fever. Today she is well and will go to school. Write a note from José to her teacher, Mr. Ivan King. Explain why Sophie was absent. (8 points)

Include an opening and a closing. Use today’s date. Write at least 2 sentences in the body of your note.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

References

Diaz, B. B., Magy, R., Salas-Isnardi, F. (2010). Future 2 English for results. White Plains, NY:

Pearson Education

Assessment Purposes

This Achievement Test is designed to cover a specific unit in an adult community-based

ESL course. Each of the three sections addresses learning outcomes for the unit. These include

differentiating between proper use of past and present tense verbs in context, reading

comprehension and scanning skills, constructing sentences with correct word order, and the

ability to write a simple absence note. Topics include U.S. holidays, and a “bad day.”

In the Grammar section, both sections A and B are assessing the test taker’s ability to

apply the correct verb tense in context. The vocabulary from these sections is all carefully

selected for appropriacy. Section C, under Reading, is assessing comprehension. The topic of

U.S. holidays was covered in class, including the vocabulary specific to the test questions. How

to scan an article for specific information was also covered and is the assessment focus of

Section D. While the topic and vocabulary should be familiar, the article includes new

information about U.S. holidays that was not included in the class so that students must read the

article to answer the questions.

The Writing section assesses several skills. The first, in section E, is the ability to apply

correct word order in a sentence with correct capitalization and ending punctuation. The

objective of Section F is to assess correct sentence word order, appropriate word choice, and the

proper use of past tense verbs. The context is a “bad day,” the vocabulary for which the students

covered in class. The final section, Section G, likewise covers the same assessment objectives,

with the addition of the conventions of an absence note.

SCORING

Grammar

Section A (6 points possible)Assessing knowledge of correct use of past or present verb tense in context. Answers 1-6. One point for each correct answer.

1. was2. declared3. became4. is5. lives6. enjoy

Section B (10 points possible)Assessing knowledge of correct use of past and present verb tense in context.Answers 7-11. Two points for each problem: one point for each correctly identified error, one point for each correct answer. No points awarded for incorrectly identified errors or incorrect answers.

7. grows, grew 8. have, has 9. marry, married10. learn, learned11. celebrate, celebrated

Reading

Section C (6 points possible)Assessing reading comprehension.Answers 12-14. One point for each correct answer.12. D13. A14. B

Section D (10 points possible)Assessing ability to scan for specific facts.Answers 15-19. One point for each correct answer. No points deducted for spelling, capitalization or punctuation errors.

15. d16. b17. a18. 3 (or “three”)19. Veteran’s Day or Memorial Day

Writing

Scoring for Section E (8 points possible)Assessing knowledge of appropriate word order in a sentence.Answers 20-23

One point for each assembled sentence that is grammatically correct. More than one correct answer may be possible.

One point for each sentence that is started with a capital letter and ended with a period. No point awarded unless both are present.

No points subtracted for spelling.

Scoring rubric for Section F and Section G (8 points possible each section)

Score Criteria

0 - No effort Response was left blank

1-5: Fails to meet expectations

Does not follow prompt directionsSevere errors in grammar, word order and formMeaning obscured or incomprehensibleDoes not use past tenseSerious errors in sentence structureRestricted vocabulary and/or inappropriate word choiceDemonstrates no mastery of conventions: note does not include opening, closing, and/or date.

6-10: Meets expectations Meets minimal prompt requirementsOccasional errors in grammar, word order and form that do not obstruct meaning.Proper use of past tense with only minimal errorsMinor errors in sentence structure.Appropriate word choice, but limited vocabulary.Demonstrates awareness of conventions, with occasional errors: note includes opening, closing and date

TOTAL POINTS POSSIBLE: 56