If you want to try out the software, go to the Knowledge Analysis Technologies
webpage at http://www.knowledge-technologies.com/, then click on the link to the
Intelligent Essay Assessor (http://www.knowledge-technologies.com/IEA.html).
I've tried out a previous product that said it graded summaries, and found that
the vaunted "latent semantic analysis" consisted of counting up keywords. The
headings and links to US Army projects in this website don't offer much hope
that the essays expected will include much critical thinking or free exploration
of ideas either.
Here's an excerpt from a press release of a few years ago (still online at
http://lsa.colorado.edu/essay_press.html). It also doesn't let the idea of
thinking interfere with the aim of measuring. Note the use of the term
"learning" to refer to the machines in the second sentence here, and the
apparent reduction of the student to "it" in the last sentence:
> First the software program is "fed" information about a topic in the form of
50,000 to 10 million words from on-line textbooks or other sources. It learns
from the text and then assigns a mathematical degree of similarity or "distance"
between the meaning of each word and any other word. This allows students to use
different words that mean the same thing and receive the same score. For
example, they could use "physician" instead of "doctor."
> The program then evaluates essays in two primary ways. The first is for a
teacher or professor to grade enough essays to provide a good statistical sample
and then use the software to grade the remainder.
> "It takes the combination of words in the student essay and computes its
similarity to the combination of words in the comparison essays," Laham said.
The student then receives the same grade as the human-graded essays to which it
is most closely matched.
Margaret Procter
University of Toronto
Tosh Tachino wrote:
> As a former (human rater) of a standardized testing, I am mildly interested in
> this issue of the ability of the computer to grade standardized essays, and
> apparently, it is making progress to the pilot stage. The following is an
> article in NY Times, describing Indiana's attempt to computerize essay grading
> in a standardized test.
>
> While reading the article, I was struck by a few things that makes me skeptical
>
> about computer grading and wondered if machines can ever really simulate (and
> eventually replace?) human raters.
>
> - It is rather troubling to find that the article discusses essay grading as a
> matter of spelling and grammar. The article seems to foreground computer's
> ability to judge grammaticality of the sentences, de-emphasizing/ignoring more
> important issues such as its ability to comprehend and respond to the content
> of the essay and evaluate the rhetorical effectiveness of the essay.
>
> - I am also concerned about the effect the machine grading might have on
> student writers whose essays will never be read by another human being. Sure,
> the students in standardized testing already do not receive human responses
> from the raters, but I fear the changes might further push the perception that
> writing is all about fixed formulae rather than a genuine human response to a
> rhetorical situation.
>
> Any thoughts on this?
>
> (Source http://www.nytimes.com/2004/05/19/education/19indiana.htm)
>
> May 19, 2004
> Indiana Essays Being Graded by Computers
> By SOL HURWITZ
>
> INDIANAPOLIS - In the computer lab at Warren Central High School in mid-May,
> Craig Butler, a junior, squinted at the question on his screen, paused to
> ponder his answer and began typing.
>
> Craig was one of 48,500 Indiana juniors gathering in high schools across the
> state to take the end-of-year online English essay test. Unlike most essay
> tests, however, this one is being graded not by a teacher but by a computer.
>
> Craig has already decided he prefers computer grading. "Teachers, you know,
> they're human, so they have to stumble around telling you what you need to do,"
>
> he said at a practice session. "A computer can put it in fine print what you
> did wrong and how to fix it."
>
> But his English teacher, Richard P. Dayment, wonders whether the computer is up
>
> to the task. "For the computer to do the subjective grading that's necessary on
>
> an essay, I'll want to see it before I have faith in it," he said.
>
> Indiana is the first state to use a computer-scored English essay test in a
> statewide assessment, and its experience could influence testing decisions in
> other states. Eighteen states now require students to pass a writing test for
> high school graduation, and, starting next year, both the SAT and the ACT will
> include writing in their college admission exams.
>
> "In five years at least 10 more states will be at or beyond the pilot stage" of
>
> automated essay scoring, predicts Richard Swartz, executive director of
> technology products and services at the Educational Testing Service, designers
> of Indiana's online essay-grading software.
>
> While Indiana's essay test is not a pass-fail "high stakes" test, it is part of
>
> an assessment of student achievement in the 40-credit state curriculum, known
> as Core 40, recommended by Indiana educators and business leaders as
> preparation for success in college and the work force. Scores on the Core 40
> tests, offered for the first time this year in English and algebra, will help
> determine college readiness and course placement for students and the
> performance ratings of high schools.
>
> With the increasing number of mandates to test student writing, "there's a
> certain inevitability to computerized essay grading," said Stan Jones,
> Indiana's commissioner of higher education. Indiana's computerized essay
> scoring, he said, will reduce by half the cost of administering a pencil-and-
> paper test and will free teachers from distributing, collecting and, above all,
>
> grading thousands of test booklets.
>
> Moreover, automated grading will yield almost instant results, allowing
> teachers to provide immediate feedback to their students. It would take weeks
> or months to receive grades on a statewide pencil-and-paper test.
>
> To dispel skepticism over computer scoring, student essays were simultaneously
> graded by a computer and trained readers during a two-year pilot program. Using
>
> artificial intelligence to mimic the grading process of human readers, the
> computer's automated scoring engine, known as e-rater, generated grades on a
> six-point scale that were virtually identical to those of the readers.
>
> Still, skepticism abounds. Although English teachers at Warren Central applaud
> the computer's ability to evaluate spelling, punctuation, grammar and
> organization, Richard C. Reed, the department chairman, made it clear that "we
> are not 100 percent sold on the computer's ability to grade content."
>
> Kathryn L. Allison, the English department chairwoman at North Central High
> School nearby, doubts that the computer can accurately assess the quality of
> grammatically correct and well-structured student essays that lack substance or
>
> are wrong on the facts. "Are kids going to be rewarded for having pedestrian-
> type answers?" she asked.
>
> Students, too, worry about the computer's accuracy.
>
> "I always wonder if, like, the computer is going to grade everything right,"
> said Jared Rampersaud, a senior at North Central, who took the test during the
> 2003 trial run, adding that "the teacher knows me and the computer doesn't."
> Jared's classmate Mollie Mott agreed. "We're always told that even the computer
>
> makes mistakes," she said. "I just think it helps if a person can actually look
>
> at" the essay.
>
> How soon other states will emulate Indiana will depend, in part, on how well
> the machine's performance compares to that of human graders. So far, pilot
> tests in Oregon, Pennsylvania, Massachusetts and South Dakota have failed to
> persuade those states to abandon human grading.
>
> Access to computers and the Internet is, of course, critical. Indiana ranks
> among the top 10 states in student access to computer technology, with one
> instructional computer for every three students, according to a report
> published in May by Education Week magazine, and all of Indiana's public high
> schools are wired for high-speed Internet.
>
> Even so, scheduling is tricky. "Let's say you've got a computer lab that's set
> up for all English classes," said Wesley D. Bruce, Indiana's director of school
>
> assessment. "You need a two-week window to test every 11th grader. So you're
> throwing 9th, 10th and 12th graders out of lab time."
>
> Technical glitches are another hurdle. During a trial run, an Internet
> configuration error prevented students at one school from submitting their
> completed essays for grading.
>
> While pleased to be in the vanguard of a technology that could transform essay
> grading nationwide, state education officials are mindful of the risks. "With
> paper and pencil we've spent decades figuring out what's going to go wrong and
> how to deal with it," Mr. Bruce said. "With online we just don't know where all
>
> the problems are."
>
> "We hope we're on the leading edge and not the bleeding edge," he said warily.
>
>
> _____________________________________
> Tosh Tachino, M.A., B.A. Honors
> Ph.D. Student, Iowa State University
> E-mail: [log in to unmask]
> http://www.public.iastate.edu/~tosh/
>
> -=-=-=-=-=-=-=-=-=-
> To leave the list, send a SIGNOFF CASLL command to
> [log in to unmask] or, if you experience difficulties,
> write to Russ Hunt at [log in to unmask]
>
> For the list archives and information about the organization,
> its newsletter, and the annual conference, go to
> http://www.stu.ca/inkshed/
> -=-=-=-=-=-=-=-=-=-
--
(Dr.) Margaret Procter
University of Toronto Coordinator, Writing Support
Room 216, University College
15 King's College Circle
Toronto, ON M5S 3H7
(416) 978-8109; FAX (416) 971-2027
[log in to unmask]
http://www.utoronto.ca/writing
-=-=-=-=-=-=-=-=-=-
To leave the list, send a SIGNOFF CASLL command to
[log in to unmask] or, if you experience difficulties,
write to Russ Hunt at [log in to unmask]
For the list archives and information about the organization,
its newsletter, and the annual conference, go to
http://www.stu.ca/inkshed/
-=-=-=-=-=-=-=-=-=-
