So we all know teachers never have enough time to implement all the awesome things they have planned. One seemingly efficient way to evaluate learners is through automated systems. State tests are all multiple choice and scantron answer documents are read and digitally recorded with very little human input. That may be great when the only possible answers are A, B, C, or D, but what happens with writing?
As an English Language Arts teacher, I dread the evenings after my students write essays because I know that I need to respond promptly and get the various stages of drafts back to my students for improvement. But what if a computer program could do it for me? Would I trust the reading of essays to algorithms? My school has a program that students can write in and hit “grade” and in a matter of seconds get a mock “TAKS” score and some stock feedback comments. They can then go back and improve their response.
Automated essay scoring systems (AES) are a type of writing assessment tools. Research on the effectiveness and validity of AES varies. Aside from the actually difference in scores given by AES and human-raters, other studies the frequency and type of feedback is compared in AES and teacher written feedback. Many studies are conducted with native-English speakers and ELLs.
This is important to anyone taking a standardized test in the near future. Could a computer be grading your essay? Do you want that? Why or why not, or does it not matter to you? Do you use programs like these? Would you? Is it okay to take the human element out for the sake of efficiency? Can these e-raters be truly valid?
There is always a level of subjectivity in grading writing. I don’t even let the school’s grader grade my students’ work.
This is a study with one e-rater program with ELLs. The literature review is of particular interest.
Automated Essay Scoring Feedback
Dikli, S. (2010) The nature of automated essay scoring feedback. CALICO Journal. 28(1), 99-134.
This comment has been removed by the author.
ReplyDeleteI don’t think I would ever use an AES. It’s a tempting idea as the amount of time it could save would be considerable. I just wouldn’t feel comfortable using it. When I grade my students’ writing, I like to take other factors into consideration, aside from grammar, mechanics and essay style. The most important thing I look for is whether or not each student’s writing ability has
ReplyDeleteimproved. This always factors into my grade. I don’t grade their work against each other, but
rather on an individual basis.
Another concern I would have about AES is how these systems rate creative writing.
Would poetic devices, humor, or other imaginative techniques be scored as adequate or
appropriate writing? This is something I doubt a computer can be programmed to perceive and
permit.
Finally, as I student, I prefer that human touch. I would not appreciate a stock answer
from a computer program. I want to hear what an actual human being has to say.
Computer essay grading is your teaching destiny!
ReplyDeleteDon't believe me?
Take a look at the latest online job applications from the government, most chain businesses, and the Accuplacer test for college.
Last summer I took a teacher workshop called "The San Antonio Writing Project" which was hosted by St. Phillips community college. I highly recommend this course for two reasons. First, you get an idea about how the Accuplacer test is scored using and automated essay scoring system and actually get to take this yourself to see what kind of score you can make based on that algorythm. You then find out that if your essay doesn't fit inside that algorythm, it gets kicked out for human review. And, although it's sort of indimidating to have a machine grade your essay, it's comforting to know that human backup is available.
My second reason for recommending this course is the fact that you get an awareness of how many students are funnelled into remediation programs in most college classrooms based on a score from this test. Many people feel that remediation is a problem that resulted from having the TAKS drive our teaching methods...but that's something for Dr. Wright to talk about. At any rate, I want to at least point out that there are computer checking systems currently running several hiring system as well. I know that online job applications have been going on in Federal government for over a decade now, and most chain businesses (Walmart, Target, etc.) all have online job forms that have computer checking systems.
From what I have been told, these computerized employment systems are set to get your application to the write person based on "trigger" words that you need to type in on either your experience or education or both. Without the skill to do this, you either don't get a job, or you don't get the job you want. My point; teach students how to read and write so that they understand how to look for key words that a computer will recognize based on the requirements for a job. Believe it or not, even if we don't like the idea of having a computer grade our essays, it seems that students preparing for the future workforce will have to gain this skill or suffer the consequences.
Regards,
Sharon
Automated Essay Scoring
ReplyDeleteA written essay reflects the writing ability of the student. If the teacher/ human grader is unable to assess the crucial writing factors including: the writer, topic, mode, time limit, etc. the main writing factors are not being assessed appropriately nor fairly. Opportunities for the scoring of essays used to measure writing ability are lost which greatly affect scores. Most of these factors are present in tests, and the person grading them, which provide a more accurate assessment in the scoring of essays. In fact, handwriting (handwriting quality, spelling) are also factors that influences the scoring of essay tests. Unfortunately, the aim of removing error factors and establishing fairness, has led many schools to use computer based automated essay scoring (AES) systems. The problem is that AES systems do not provide the users with useful feedback on their writing errors and ways to improve their writings. Limited conclusions can be provided from the graded writing. AES systems are mainly used to overcome time, cost, and reliability. As a result of the wide use of AES, writing teachers may lose control of the writing construct as what they dictate/teach is graded by a machine.
I would not rely on a machine to make critical decisions for my students. For some assignments, it could be useful, but not for writing. This is an area of communication where teacher and student interact to achieve writing goals, and a machine simply does not provide this privilege.
This is only the second time I have heard about computerized essay grading with regards to language teaching, so pardon my ignorance on the subject. However, after reading the abstract of the article in CALICO, my inclination at the moment is that for language learners, essays should be graded by teachers who can give salient, directed, individualized feedback at a level that the ELL can understand. Grammar can be checked and some syntax corrected by a computer, but wouldn’t things like style and developing an argument still need a teachers’ appraisal? If the teacher still has to read the essay, how much time would be saved?
ReplyDeleteAlthough it might be, and probably is, more efficient to grade masses of essays using the AES, in my opinion, the classroom teacher’s own thoughtful remarks and feedback stand much more of a chance of being read, thought about, remembered and appropriated in future than does the automated generic feedback that is provided in this type of system. Too, if the teacher does not personally do the grading, how can he/she keep up with understanding the exact errors and nature of the problems that need to be addressed (except on a very general basis)?
As for students using the system for individual work, I would have to see this in action to assess if it could be useful in the initial stages of writing an essay, or even too helpful as a spellcheck program sometimes is!
Again, in writing an essay, a student is motivated in part by addressing a human audience and by displaying their own creativity. If a student has to instead attend to working in some key words and phrases that the computer will count as being up to standard, his motivation and interest could be hampered.
This brings to mind the point of quality of learning and the learning environment. It could be that AES is the future and will be useful in some instances and I would welcome learning about this, but there may also be drawbacks involved. Language learning is one area where many factors are in play such as cultural understanding and adaptation, growing confidence in using another language and ELLs learning how to express themselves in new ways. Corrections of errors can be a sensitive business and computers aren’t known for their understanding. Perhaps not all errors should be addressed all the time and presumably the teacher would be the best judge of that.
I remember I used a kind of automatic essay scoring software while I prepared for the GRE essay in Taiwan. You need to pay 30 dollars to get access to the website and you can use that system for 3 months. To tell the truth, it's not a good software for me at all. First of all, it did not provide useful feedback for my writing. It would give you comments like you cannot get high scores because the paragraph is not organized. For language learners, it would be good to get specific feedback about my essay, including grammatical errors and how I can improve my writing skills. Secondly, I prefer face-to-face conversation instead of receiving messages from the computer. I might have extra questions about writing an essay and I can get answers immediately from personal editor.
ReplyDeleteIf I work in the school with automatic writing system, I will choose to use e-writer and personal editer at the same time. It can be effective to save teachers' time in grading students' papers. I want to grade their paper personaly because I can see individual problems in learning the target language and give them advice to improve their language proficiency level. It migth be more useful for students in learning language. Also, students can negotiate meaning and it can push them to have comprensible output through communication. That's why I think the human element cannot be ignored from language learning.
Automatic essay grading seems like the new age technology and I do see points to both sides of the spectrum discussed above. Yes, I see the validity of Sharon's argument that AES is the future and it doesn't matter how we feel about AES, it is here to stay. Wow, pretty powerful stuff! I love technology, especially any that will free up my hands long enough plan more lessons. I have never used this type of program before, but I would be open to testing it in my classroom. Plus, I want my students to be able to land a job by knowing how technology works in their community. Before I would be able to instill my trust in this program I would have to test it out. I would run all sorts of papers through and who knows maybe the program would be able to produce effective feedback. I am open to trying AES in my classroom, as an additional tool to help my students prepare for the future.
ReplyDeleteAnother perspective mentioned above is the fact that AES should not be used for language learning students. Susan had a great point, and I agree that AES can not be the end-all, solve-all program. While AES might be satisfactory to native english speakers, it may be cruel and unusual punishment to allow a computer to grade students that are learning the language as a second language. We can not allow our writers to be programmed to a specific formula. Our feedback and personality expressed to language learners can be the light at the end of a tunnel that turns on as language learners understand written expression.
Students who take the time to write deserve a teacher willing to read their work. I believe there is a balance between AES and direct feedback from a person that knows the student and understands what the student is capable of. I believe a good teacher would implement the AES program as a tool in helping the students become even better writers by incorporating the "rules" of scoring by AES into their natural written expression.
This is a particularly timely topic in light of a recent research manuscript I reviewed. The author had attempted to compare the feedback accuracy as well as the learning and revision delivered by a classroom teacher with that delivered by a piece of writing software.
ReplyDeleteThe author found that the software was less accurate than the teacher, particularly when it came to discourse level features (i.e. writing issues that extend beyond the sentence level) and would often over correct certain issues that were not errors (e.g. use of passive) and miss other errors that were incorrect in context (e.g. some possessive constructions).
As a result of the inconsistency in the feedback from the computer, students were less likely to correct their errors in revisions of drafts submitted to the computer program than to the teacher. Student opinion on the software indicated that they found the teacher better at helping them with their writing than the software.
Although not directly addressing the issue of assessment, this study did highlight some of the aspects of writing that are difficult for computer applications to catch.
I think my opinion has been expressed pretty well by others who have posted...but just to restate..
ReplyDeleteI would not feel comfortable handing over my review of student essays/writing to a AES type program. It would be awesome if this type of rater could indeed support student writing in the same way as human teacher, but knowing the little I do about these systems and after reading the posted article, it seems clear that the AES systems are not quite ready to take over with effective writing feedback, correction, encouragement, etc. There is more to writing than the use of passive sentences, mixed up grammar, misspellings and the types of errors that computers can check for. Maybe their writing has a poetic element that can be commented on or etc...an element that could even be marked as an error by the system because it doesn't fit into a specific algorithm.
However, it is also true that many companies are using these types of systems and have done so for years. I think it's a shame because you could miss out on a really great employee who may not fit the exact mold you designed but has other attributes that would benefit the company. Without the human element, slightly out of the box potential employees may not be hired...and they are often the best sort! BUT...they are being used and our students will no doubt come in contact with them if only during a job search. For this reason, it could be a great idea to have a writing class, or at least part of some writing classes, touch on how to write to score well on AES systems. I think this could especially apply when it comes to resumes and job hunting. It's a skill like any other that can be taught--kind of like taking a TAKS test, the SAT, or TOEFL. We all had to learn the tricks.
Right now, I think that the whole idea of AES is a little too much. I remember a good example that I believe one of the students in our class, who had to take the ESL entrance exam, like TOFEL or something like that, gave. She said that her teacher told her to write one grammatical sentence, and then do it over and over again, using the 5 paragraph format. The machine that graded the essay graded not language diversity, but rather grammatical correctness. As a result the student got an easy A. However, if anyone has been watching any commercials lately you might have seen the IBM commercial about a super computer that can play Jeopardy. Now that machine can get a shot at grading my essay. It sounds like soon computers will be able to do a good job doing the grading for teachers, but for now I don't think there is a good alternative to two sets of eyes and a brain.
ReplyDeleteThe only reason I would want a computer to grade my essay is for it to act as a reference. At the same time, I would want to know what the computer is looking for, such as certain sentence structures, vocabulary, etc. This is because whoever created the software to look for this features must have taken these elements seriously. In Taiwan, cram schools exist to help students pass tests, any kinds of tests (TOEFL, TOEIC, SAT, GMAT, GRE, wow, I took them all) and most cram schools will provide you with a magical template that will help you pass your written test. Your score may not be sky-high but you will pass with an average score, which in a lot of cases, is enough for beginning/intermediate English language learners.
ReplyDeleteHowever, as a student, I always love getting my paper back with comments all over it. To me, it means that the grader/professor has put in time and effort to read over it and also, it is way for communicating thoughts and ideas. Learning how to write is not just about typing out perfect, grammatical sentences! That is why Dell Hymes, Canale and Swain came up with other competence (Had to throw these in, I’m preparing for comps). I would want to see my students challenge themselves and make errors, to take risk and to be able to convey their thoughts through writing awesome essays. Perhaps when it comes to taking a test, my students will have to be careful and not make any mistakes but when they are just practicing, I will want to understand them more, know their strengths and weaknesses and provide them with mini lessons to improve their writing. That being said, I personally don’t think we should take the human factor out when it comes to grading papers.
@Angie, I am always happy to see a reference to Canale and Swain crop up in a CALL discussion.
ReplyDelete@ZweiRu, your example illustrates the need for a much more sophisticated computer scoring system than that which a lot of institutions can afford if the goal is to teach students language and not just how to game the system.
Automated testing of multiple choice is one thing, but automated testing for complex content, or essays is a completely different topic to which one answer is quite sufficiency. Computers are not intelligent: they are strict and only operate within a very limited scope. Essays are creative, and spontaneous. Each student has their own unique and exceptional style. What computers have done for us is quite appalling, they have given us a way to keep everyone within the strict guidelines. This will only get worse. There was once this ida that computer would get more sophisticated, and as a programmer, I know that instead the industry its-self has simply become more fractured. Computers are not more sophisticated than they were 20 years ago. Computers are good at ding one-one tasks (which is the nature of multiple choice, solving an equation, giving results based on a search query, but at no point does the computer notice linguistic poetry, that would require understanding).
ReplyDeleteFinely, I agree, one should not rely on computers to make any decision, computers are tools to give you an idea of where you should go, to help show you a trend, but even for decisions that are less than 'critical' the computer has no business in making decisions.
I really do not think that I would prefer to use the Automated Easy Scoring with my students, because I strongly believe using this kind of softwares to evaluate your students’ writing is not fair at all.
ReplyDeleteWe know AES saves much time for teachers, but it will destroys students’ futures. Students need real feedback from their teachers and enough explanations.
Finally, I would like to say that it is not going to happen that the software will evaluate students’ writing better than teachers.
Regards
Automated Essey Scoring could be in great help for teachers. Those, who used to slave with paperwork, might agree with me. Too much duties and responsibilities, but after all you want to be good teacher and be sure you are in right place in right time.If some esseys can be scored with no my assistance and help my student to upgrade his writing skills - I will take care of other 50% of this job. This way I would have more time for myself and feel much more comfortable and valuable.
ReplyDelete