Re: Ideas
I'm not sure how a computer can mark originality
First, some assumptions: The subject matter is constrained and not esoteric. The writers are not subject-matter experts or expert writers. The evaluation system has a large corpus of work on the subject. We're interested in conceptual originality, not, say, originality of style.
Then as long as you have a decent semantic-content extraction algorithm, you just create a model of the topic based on your corpus, and then for each candidate essay compute a series of metrics in increasing dimensionality: How far does the candidate differ from the first-order semantic content (individual ideas, very roughly speaking)? How far does it differ in second-order semantic content (ideas paired by a relation)? In third-order (networks with three significant concepts)?
Doing that even for third-order relationships, in this kind of constrained environment, is likely to give you an "originality" metric that nearly always corresponds closely with how human expert judges would rate the candidates, and is relatively difficult to game.
There are other abstract attributes which are much more difficult to reduce to this sort of model-correspondence, however. Beauty (aesthetic appeal of the prose style, imagery, etc) is one. Logical consistency is another, as is psychological consistency. (Those two require sophisticated world models which have historically proven very tough to build.)
In other words, it's not that hard to mechanically analyze what a sample of prose is. It's hard to analyze what it does for a human audience. But the beauty of standardized tests, for the people who want to see this kind of tech, is that they don't want to measure the latter, because it's too subjective.
Biting the hand that feeds IT
