As a former (human rater) of a standardized testing, I am mildly interested in
this issue of the ability of the computer to grade standardized essays, and
apparently, it is making progress to the pilot stage. The following is an
article in NY Times, describing Indiana's attempt to computerize essay grading
in a standardized test.
While reading the article, I was struck by a few things that makes me skeptical
about computer grading and wondered if machines can ever really simulate (and
eventually replace?) human raters.
- It is rather troubling to find that the article discusses essay grading as a
matter of spelling and grammar. The article seems to foreground computer's
ability to judge grammaticality of the sentences, de-emphasizing/ignoring more
important issues such as its ability to comprehend and respond to the content
of the essay and evaluate the rhetorical effectiveness of the essay.
- I am also concerned about the effect the machine grading might have on
student writers whose essays will never be read by another human being. Sure,
the students in standardized testing already do not receive human responses
from the raters, but I fear the changes might further push the perception that
writing is all about fixed formulae rather than a genuine human response to a
rhetorical situation.
Any thoughts on this?
(Source http://www.nytimes.com/2004/05/19/education/19indiana.htm)
May 19, 2004
Indiana Essays Being Graded by Computers
By SOL HURWITZ
INDIANAPOLIS - In the computer lab at Warren Central High School in mid-May,
Craig Butler, a junior, squinted at the question on his screen, paused to
ponder his answer and began typing.
Craig was one of 48,500 Indiana juniors gathering in high schools across the
state to take the end-of-year online English essay test. Unlike most essay
tests, however, this one is being graded not by a teacher but by a computer.
Craig has already decided he prefers computer grading. "Teachers, you know,
they're human, so they have to stumble around telling you what you need to do,"
he said at a practice session. "A computer can put it in fine print what you
did wrong and how to fix it."
But his English teacher, Richard P. Dayment, wonders whether the computer is up
to the task. "For the computer to do the subjective grading that's necessary on
an essay, I'll want to see it before I have faith in it," he said.
Indiana is the first state to use a computer-scored English essay test in a
statewide assessment, and its experience could influence testing decisions in
other states. Eighteen states now require students to pass a writing test for
high school graduation, and, starting next year, both the SAT and the ACT will
include writing in their college admission exams.
"In five years at least 10 more states will be at or beyond the pilot stage" of
automated essay scoring, predicts Richard Swartz, executive director of
technology products and services at the Educational Testing Service, designers
of Indiana's online essay-grading software.
While Indiana's essay test is not a pass-fail "high stakes" test, it is part of
an assessment of student achievement in the 40-credit state curriculum, known
as Core 40, recommended by Indiana educators and business leaders as
preparation for success in college and the work force. Scores on the Core 40
tests, offered for the first time this year in English and algebra, will help
determine college readiness and course placement for students and the
performance ratings of high schools.
With the increasing number of mandates to test student writing, "there's a
certain inevitability to computerized essay grading," said Stan Jones,
Indiana's commissioner of higher education. Indiana's computerized essay
scoring, he said, will reduce by half the cost of administering a pencil-and-
paper test and will free teachers from distributing, collecting and, above all,
grading thousands of test booklets.
Moreover, automated grading will yield almost instant results, allowing
teachers to provide immediate feedback to their students. It would take weeks
or months to receive grades on a statewide pencil-and-paper test.
To dispel skepticism over computer scoring, student essays were simultaneously
graded by a computer and trained readers during a two-year pilot program. Using
artificial intelligence to mimic the grading process of human readers, the
computer's automated scoring engine, known as e-rater, generated grades on a
six-point scale that were virtually identical to those of the readers.
Still, skepticism abounds. Although English teachers at Warren Central applaud
the computer's ability to evaluate spelling, punctuation, grammar and
organization, Richard C. Reed, the department chairman, made it clear that "we
are not 100 percent sold on the computer's ability to grade content."
Kathryn L. Allison, the English department chairwoman at North Central High
School nearby, doubts that the computer can accurately assess the quality of
grammatically correct and well-structured student essays that lack substance or
are wrong on the facts. "Are kids going to be rewarded for having pedestrian-
type answers?" she asked.
Students, too, worry about the computer's accuracy.
"I always wonder if, like, the computer is going to grade everything right,"
said Jared Rampersaud, a senior at North Central, who took the test during the
2003 trial run, adding that "the teacher knows me and the computer doesn't."
Jared's classmate Mollie Mott agreed. "We're always told that even the computer
makes mistakes," she said. "I just think it helps if a person can actually look
at" the essay.
How soon other states will emulate Indiana will depend, in part, on how well
the machine's performance compares to that of human graders. So far, pilot
tests in Oregon, Pennsylvania, Massachusetts and South Dakota have failed to
persuade those states to abandon human grading.
Access to computers and the Internet is, of course, critical. Indiana ranks
among the top 10 states in student access to computer technology, with one
instructional computer for every three students, according to a report
published in May by Education Week magazine, and all of Indiana's public high
schools are wired for high-speed Internet.
Even so, scheduling is tricky. "Let's say you've got a computer lab that's set
up for all English classes," said Wesley D. Bruce, Indiana's director of school
assessment. "You need a two-week window to test every 11th grader. So you're
throwing 9th, 10th and 12th graders out of lab time."
Technical glitches are another hurdle. During a trial run, an Internet
configuration error prevented students at one school from submitting their
completed essays for grading.
While pleased to be in the vanguard of a technology that could transform essay
grading nationwide, state education officials are mindful of the risks. "With
paper and pencil we've spent decades figuring out what's going to go wrong and
how to deal with it," Mr. Bruce said. "With online we just don't know where all
the problems are."
"We hope we're on the leading edge and not the bleeding edge," he said warily.
_____________________________________
Tosh Tachino, M.A., B.A. Honors
Ph.D. Student, Iowa State University
E-mail: [log in to unmask]
http://www.public.iastate.edu/~tosh/
-=-=-=-=-=-=-=-=-=-
To leave the list, send a SIGNOFF CASLL command to
[log in to unmask] or, if you experience difficulties,
write to Russ Hunt at [log in to unmask]
For the list archives and information about the organization,
its newsletter, and the annual conference, go to
http://www.stu.ca/inkshed/
-=-=-=-=-=-=-=-=-=-
