the online writing evaluation system panacea: myth or reality?

Susan BleyleSemire Dikli

Georgia Gwinnett College

SETESOL 2009

Main purpose What is OWE?

Advantages Disadvantages

Problem statementProcess approach requires a great amount

of time and effort from the teachers.Feedback on writing has been found to be

highly individual and/or situation specific.Many OWE systems were originally

designed for native English speakers, but they were developed and marketed for non-native speakers, as well.

What are the student perceptions of using an OWE system in a college ESL classroom?

15 advanced ESL students in a writing course for matriculated studentsprimarily U.S. high school graduates from various linguistic backgrounds

Student drafts Survey

Over the course of a semester, students wrote a minimum of two drafts on six prompts3 were written using an OWE system. 3 were written in-class using pen and paper

and revised at home using Microsoft Word. • At the end of the semester, students

completed a survey to ascertain their perceptions of using OWE as compared to the more traditional essay formats

Survey results are tallied:Planning WritingFeedbackScoringRevising

Do you plan your essays (e.g. brainstorm, map, outline) before writing?

1. Never 02. Rarely 2 (14.3%)3. Usually 5 (35.7%)4. Always 7 (50%)

Which do you generally prefer for planning your essays?

1. Criterion 2 (14.3%)2. Paper and pencil/pen 11 (78.6%)3. Other 2 (14.3%)

Negative comments:“I prefer planning my essay on paper because I can see my ideas in an expanded way.”

“I like to use paper and pen for planning my essay. I feel comfortable writing on the paper better than on computer.”

Neutral comments: “I have never use the “Make a Plan” feature, so I do not know what I don’t like about it.”

Which do you generally prefer for writing your essays?

1. Criterion 5 (35.7%)2. Paper and pencil/pen 7 (50%)3. Other 4 (28.6%)

Postive comments:“I like Criterion better because it has spell check and sometimes I'm not sure how to spell words. Also, I like Criterion because I'm saving paper.”

“I prefer both Criterion and on paper because I like to brainstorm on paper before writing it on Criterion.”

Negative comments:“I like to use paper and pen writing essay better than use Criterion. Because I turn on the computer that make some noise, I cannot concentrate to plan my essay.”

Other:“I like use Microsoft word because it can correct some mistakes and spelling. Furthermore, it also have many kind to choose to make my essays more interesting such as color, frame, art...”

How helpful do you find the Instructor Pop-Up Notes on Criterion?

1. Not at all helpful 1(7.1%)

2. Not very helpful 1 (7.1%)3. Somewhat helpful 7

(50%)4. Very helpful 6

(42%)

How helpful do you find the Instructor General Comments on Criterion?

1. Not at all helpful 02. Not very helpful 1 (7.1%)3. Somewhat helpful 5 (35.7%)4. Very helpful 8 (57.1%)

How helpful do you find the Criterion Trait Feedback (feedback on grammar, mechanics, usage, style, etc.)?

1. Not at all helpful 2 (14.3%)2. Not very helpful 2 (14.3%)3. Somewhat helpful 7 (50%)4. Very helpful 4 (21.4%)

How do you prefer getting feedback on your essays?

1. On Criterion 5 (38.5%)2. On paper 8 (61.5%)3. Other 2 (15.4%)

Positive comments:

“i like the criterion feedback and the instructor feedback because both helped alot in rewriting my last draft”

“Because it is easier for me to see my mistakes on Criterion.”

Negative comments:

“Criterion is not 100% correct. Sometimes, it would give false feedback.”

“not helpfull because it does not read the run on or other misstakes that only a profesor can faind.”

“Either one would work, but on Criterion its not as clear as on paper.”

How helpful do you find the Criterion Essay Score of 1-6?

1. Not at all helpful 2(14.3%)

2. Not very helpful 3 (21.4%)3. Somewhat helpful 9 (64.3%)4. Very helpful 0

Comments:

“I like that it helps us become better writers. What I don't like is that sometimes it'll rate the essay good but deep down inside I know it's not the best that i can produce. Also, it pinpoints any error and sometimes there is no error to be fixed.”

“I don't like the way it work with the scoring... But i like that it gives a lor of feed back.”

Which way do you prefer to revise your essays?

1. On Criterion 4 (33.3%)2. On Paper 5 (41.7%)3. Other 4 (33.3%)

Positive Comments:

“I like it [Criterion] because I don't have to rewrite my entire essay, I can just correct it.”

“I think it is easier to revise my essay on Criterion because I can see my teacher's grammar comments clearly.”

Negative Comments:

“I like to use words to revise my essay. Because words format can save a copy and easy to correct or edit. However, Criterion format is not many function, for example, it can not enlarge the character.”

“I like doing it on paper or on the microsoft word make ir easier to read and understand.”

1. That Criterion be used in the same way in future semesters

2. That Criterion be used in future semesters with some changes (please explain below)

3. That Criterion be made optional in future semesters

4. That Criterion not be used at all in future semesters

5 (45.5%)

3 (27.3%)

3 (27.3%)

1 (9.1%)

Which of the following is your overall recommendation for Criterion:

Negative comments:

“I think we don't need use criterion. It's nothelp somebody to improve their essays.”

“many student can fail because of thatsoftware”

Positive comments:

“the criterion is really a good resource, so it has to be use in future too.”

“I think Criterion is a good way to practice writing essay. Sometimes, Criterion feedback is not 100% true, but I can get a hint of how I did on my essays.”

Suggestions for change:

“Let the students decide if they would like to submit their thruough criterion or paper”

“I think the Criterion format should be changed at least as same as Microsoft word's function. Because Criterion program function is limited. For example, essay cannot post double line in row, enlarge character,underline ,and add different color.”

For schoolsBudget Classroom resources

For teachersAs a support tool Inform students of limitations

For developersMore user friendlyMore attention to ESL writers

Contact information:

Susan Bleyle Semire Diklisbleyle@ggc.edu sdikli@ggc.edu