Whenever I read of the increasing use of computers to assess student writing, I
always think of the huge outcry that would meet a proposal to grade student essays
in, say, Social Studies or History by computer. Why would such a proposal be
"thinkable" in the context of writing assessment but unthinkable in the context of
a content-area course? How have we as writing teachers (and our predecessors)
contributed to a larger culture in which computerized assessment of writing is not
only considered possible but also accepted as reasonable? It appears that our
perceived preoccupations with grammatical correctness have come to haunt us...
Jo-Anne Andre
U of Calgary
Tosh Tachino wrote:
> As a former (human rater) of a standardized testing, I am mildly interested in
> this issue of the ability of the computer to grade standardized essays, and
> apparently, it is making progress to the pilot stage. The following is an
> article in NY Times, describing Indiana's attempt to computerize essay grading
> in a standardized test.
>
> While reading the article, I was struck by a few things that makes me skeptical
>
> about computer grading and wondered if machines can ever really simulate (and
> eventually replace?) human raters.
>
> - It is rather troubling to find that the article discusses essay grading as a
> matter of spelling and grammar. The article seems to foreground computer's
> ability to judge grammaticality of the sentences, de-emphasizing/ignoring more
> important issues such as its ability to comprehend and respond to the content
> of the essay and evaluate the rhetorical effectiveness of the essay.
>
> - I am also concerned about the effect the machine grading might have on
> student writers whose essays will never be read by another human being. Sure,
> the students in standardized testing already do not receive human responses
> from the raters, but I fear the changes might further push the perception that
> writing is all about fixed formulae rather than a genuine human response to a
> rhetorical situation.
>
> Any thoughts on this?
>
> (Source http://www.nytimes.com/2004/05/19/education/19indiana.htm)
>
> May 19, 2004
> Indiana Essays Being Graded by Computers
> By SOL HURWITZ
>
> INDIANAPOLIS - In the computer lab at Warren Central High School in mid-May,
> Craig Butler, a junior, squinted at the question on his screen, paused to
> ponder his answer and began typing.
>
> Craig was one of 48,500 Indiana juniors gathering in high schools across the
> state to take the end-of-year online English essay test. Unlike most essay
> tests, however, this one is being graded not by a teacher but by a computer.
>
> Craig has already decided he prefers computer grading. "Teachers, you know,
> they're human, so they have to stumble around telling you what you need to do,"
>
> he said at a practice session. "A computer can put it in fine print what you
> did wrong and how to fix it."
>
> But his English teacher, Richard P. Dayment, wonders whether the computer is up
>
> to the task. "For the computer to do the subjective grading that's necessary on
>
> an essay, I'll want to see it before I have faith in it," he said.
>
> Indiana is the first state to use a computer-scored English essay test in a
> statewide assessment, and its experience could influence testing decisions in
> other states. Eighteen states now require students to pass a writing test for
> high school graduation, and, starting next year, both the SAT and the ACT will
> include writing in their college admission exams.
>
> "In five years at least 10 more states will be at or beyond the pilot stage" of
>
> automated essay scoring, predicts Richard Swartz, executive director of
> technology products and services at the Educational Testing Service, designers
> of Indiana's online essay-grading software.
>
> While Indiana's essay test is not a pass-fail "high stakes" test, it is part of
>
> an assessment of student achievement in the 40-credit state curriculum, known
> as Core 40, recommended by Indiana educators and business leaders as
> preparation for success in college and the work force. Scores on the Core 40
> tests, offered for the first time this year in English and algebra, will help
> determine college readiness and course placement for students and the
> performance ratings of high schools.
>
> With the increasing number of mandates to test student writing, "there's a
> certain inevitability to computerized essay grading," said Stan Jones,
> Indiana's commissioner of higher education. Indiana's computerized essay
> scoring, he said, will reduce by half the cost of administering a pencil-and-
> paper test and will free teachers from distributing, collecting and, above all,
>
> grading thousands of test booklets.
>
> Moreover, automated grading will yield almost instant results, allowing
> teachers to provide immediate feedback to their students. It would take weeks
> or months to receive grades on a statewide pencil-and-paper test.
>
> To dispel skepticism over computer scoring, student essays were simultaneously
> graded by a computer and trained readers during a two-year pilot program. Using
>
> artificial intelligence to mimic the grading process of human readers, the
> computer's automated scoring engine, known as e-rater, generated grades on a
> six-point scale that were virtually identical to those of the readers.
>
> Still, skepticism abounds. Although English teachers at Warren Central applaud
> the computer's ability to evaluate spelling, punctuation, grammar and
> organization, Richard C. Reed, the department chairman, made it clear that "we
> are not 100 percent sold on the computer's ability to grade content."
>
> Kathryn L. Allison, the English department chairwoman at North Central High
> School nearby, doubts that the computer can accurately assess the quality of
> grammatically correct and well-structured student essays that lack substance or
>
> are wrong on the facts. "Are kids going to be rewarded for having pedestrian-
> type answers?" she asked.
>
> Students, too, worry about the computer's accuracy.
>
> "I always wonder if, like, the computer is going to grade everything right,"
> said Jared Rampersaud, a senior at North Central, who took the test during the
> 2003 trial run, adding that "the teacher knows me and the computer doesn't."
> Jared's classmate Mollie Mott agreed. "We're always told that even the computer
>
> makes mistakes," she said. "I just think it helps if a person can actually look
>
> at" the essay.
>
> How soon other states will emulate Indiana will depend, in part, on how well
> the machine's performance compares to that of human graders. So far, pilot
> tests in Oregon, Pennsylvania, Massachusetts and South Dakota have failed to
> persuade those states to abandon human grading.
>
> Access to computers and the Internet is, of course, critical. Indiana ranks
> among the top 10 states in student access to computer technology, with one
> instructional computer for every three students, according to a report
> published in May by Education Week magazine, and all of Indiana's public high
> schools are wired for high-speed Internet.
>
> Even so, scheduling is tricky. "Let's say you've got a computer lab that's set
> up for all English classes," said Wesley D. Bruce, Indiana's director of school
>
> assessment. "You need a two-week window to test every 11th grader. So you're
> throwing 9th, 10th and 12th graders out of lab time."
>
> Technical glitches are another hurdle. During a trial run, an Internet
> configuration error prevented students at one school from submitting their
> completed essays for grading.
>
> While pleased to be in the vanguard of a technology that could transform essay
> grading nationwide, state education officials are mindful of the risks. "With
> paper and pencil we've spent decades figuring out what's going to go wrong and
> how to deal with it," Mr. Bruce said. "With online we just don't know where all
>
> the problems are."
>
> "We hope we're on the leading edge and not the bleeding edge," he said warily.
>
> _____________________________________
> Tosh Tachino, M.A., B.A. Honors
> Ph.D. Student, Iowa State University
> E-mail: [log in to unmask]
> http://www.public.iastate.edu/~tosh/
>
> -=-=-=-=-=-=-=-=-=-
> To leave the list, send a SIGNOFF CASLL command to
> [log in to unmask] or, if you experience difficulties,
> write to Russ Hunt at [log in to unmask]
>
> For the list archives and information about the organization,
> its newsletter, and the annual conference, go to
> http://www.stu.ca/inkshed/
> -=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-
To leave the list, send a SIGNOFF CASLL command to
[log in to unmask] or, if you experience difficulties,
write to Russ Hunt at [log in to unmask]
For the list archives and information about the organization,
its newsletter, and the annual conference, go to
http://www.stu.ca/inkshed/
-=-=-=-=-=-=-=-=-=-
