AWA: Computers don't know jack about wit

[HopefulGrad2B]

Hey everyone! I recently took the GRE. I'm actually still waiting on my AWA score. I'm not placing much importance on it though, especially after reading that the essay is scored in part by "e-raters" or computer readers. I wrote (what I thought) was a creative and attention-grabbing introduction for my Issue essay and I found that both of my essays read well rhetorically and made good use of GRE vocabulary. I'm now wondering whether all of that was for naught, given the way the essays are graded. Here's a simplified explanation along with some articles with further information. Essentially you have a human reader and a bot reader. An essay that a human reader might praise for rhetorical finesse or incisiveness might be completely lost on the bot which is programmed to look for very specific features. Apparently if you misspell certain transition words (e.g. "therefore," "furthermore," etc.) the program erroneously penalizes your essay for lack of structure. The human grader is pressured to assign very strict scores using the most rigid and trite templates as guidelines (e.g. the 5 paragraph high school essay) because sincere scoring would call their work into question (i.e. if they constantly disagree with the computer). This would necessitate too much intervention from "third" (read: human) readers, which is costly to ETS. I think it's a shame that this is being done mostly (if not, entirely) for cost-cutting reasons (don't we pay ETS enough... and indeed, overpay them?). These essays are supposed to measure our analytical reasoning and an algorithm might be able to pinpoint some classical signs of good reasoning and structure but with the consequence of completely eschewing creativity and the kind of original writing many of us will go on to do in graduate school, it's no wonder a lot of humanities programs accord little importance to the AWA score.

http://awatips.blogspot.com/2007/05/using-strategy.html (This is about the GMAT but apparently it's applicable to the GRE as well)

http://awatips.blogspot.com/2007/05/about-e-rater.html (Ditto)

http://www.800score.com/gmat-essay.html (Ditto)

http://www.daytondailynews.com/project/content/project/tests/0524testautoscore.html

Since score recipients receive copies of our AWA essays, they would be better judges than the e-rater as to the quality of our writing. However, as it has often been stressed on this site, a solid writing sample and SOP are clearly better examples of one's writing and argumentation skills. I scored 800 on the Verbal section and I suspect that other 800 scores and some folks with 700+ scores might find that a score below a 5 on the AWA is a blow to our otherwise impressive GRE scores. I've encountered a plethora of threads initiated by high Verbal scorers who seem devastated about receiving a low AWA score. I don't know my AWA score yet but I'm sure I didn't do everything I could have to "please" the bot reader, so I personally won't be too disappointed if it's below a 5 and I certainly won't take it as an indication of my writing abilities. I hope this thread is helpful to those of you who (especially in the humanities) have aced the Verbal but not the AWA or who have yet to take the exam. Don't worry about the AWA; knock 'em dead with your writing sample and SOP!

[HopefulGrad2B]

P.S. Ignore the unnecessary possessive in the thread title. It was a little mistake that unfortunately I can't edit.

[eklavya]

hehe, another fired up rant! against ETS! this is gold

[HopefulGrad2B]

hehe, another fired up rant! against ETS! this is gold

Haha, thanks, I guess? I was hoping it would be more of an informative post rather than just a rant. I hope my post comes across as neither saturnine nor vituperative like the preponderance of AWA threads lately. I'm hoping that the people who typically write those types of rants will put their AWA scores in perspective and realize that the grading system is completely and utterly perfunctory.

[waddle]

Props to an informative summary!

Since score recipients receive copies of our AWA essays, they would be better judges than the e-rater as to the quality of our writing.

I don't recall having heard about this from ETS when I took the GRE a few months ago, and have had a hard time finding info about whether the essays are actually sent to score recipients (I've seen it tossed around on the GradCafe fora, but haven't found anything from ETS about this). Could you point me to where I could find info on this?

[...] the grading system is completely and utterly perfunctory.

But hey, same thing for most of the essays you write for your undergraduate career, 'cept you just have four years' worth of scores based on cursory glances, rather than four hours' worth.

Edited November 30, 2010 by waddle

[HopefulGrad2B]

Hey Waddle, thanks! I'm glad you found it helpful.

I'll admit that I'm not 100% certain about whether adcomms actually do receive our essays. I know that's how it worked with the SAT (they used to scan the handwritten essays) and I imagine it would only be simpler to do it with the current computer-based format. Then again, ETS and the Collegeboard are separate entities. I'd be curious if anyone could definitively settle this question.

As for perfunctory grading, in my case, I almost always got very thorough feedback on my writing in my undergraduate classes, but perhaps I lucked out and had really conscientious professors...

[shepardn7]

Well, I feel similarly angry and frustrated about the entire ETS corporation, and the GRE test in particular. Honestly, I find the verbal section even more worthless than the AWA section, because of its arbitrariness (how, for example, taking a different test can mean a very different score) and the particular skill set required to excel in it. I knew every single word on the test, performed 40 points better on Powerprep, and yet I still got analogies and antonyms wrong, likely because I wasn't thinking "like them" instinctually, at least not at the "higher" levels. The timer ticking down freaked me out and caused me to rush through questions could have considered further, and possibly gotten correct. I didn't do poorly; I scored in the 97th percentile in the verbal and 94th in the writing (but I wrote "stock" essays on purpose, knowing the system beforehand! I'm in creative writing, but I wrote the most boring essays of my life). But that I feel terrible about my 97th not being 99th (like your amazing score!) speaks volumes about this test, and how it causes so much unnecessary anxiety while simultaneously being a relatively worthless indicator of potential. I've never felt so inadequate about my performance on something (with the exception of the SAT, maybe, and that one paper in college that I messed up), and yet I do not even consider the test valid. Why should I care this much about my perfectly "acceptable" 97th percentile score when every neuron firing in my brain finds gaping flaws in the test's ability to assess my (or anyone's) intellectual readiness for graduate school in the humanities? And yet, here I am, feeling sad I didn't get a 750V. Sigh. All I know: if I were on an admissions committee for the humanities, I would not count the test at all, because I find it inaccurate at best, and unethical at worst. /End rant!

[gradschoolorbust!]

sarandipity, I felt that way once:

I'm over it now. Seriously, we're both in a handful of the *highest* Verbal scorers worldwide!! Above a 700, the 10 point distinctions mostly reflect test-taking stamina and savvy rather than actual knowledge of the vocabulary. I was hoping for a 700 for the psychological impact it would have over a nearly as impressive score in the high 600s (if anything because of the "7" in front)... similar to in marketing when prices are 1 cent less, i.e. 699.99 just so you don't choke upon seeing the 7 (but obviously this works in the opposite direction). Ok, maybe I'm not 100% totally over it. Like you though, I was aware of the grading system for the AWA and also wrote something more on the prosaic side. What a worthless test. I'm sure the adcomms will like the 720 (which is above all the average scores in every department I'm considering)... Even if I could score a little higher on a re-test, there is no way I would give ETS the pleasure of an additional $160. What a pointless waste of money that would be... /rant

Anyway, props to the OP for a great thread!

[shepardn7]

sarandipity, I felt that way once:

I'm over it now. Seriously, we're both in a handful of the *highest* Verbal scorers worldwide!! Above a 700, the 10 point distinctions mostly reflect test-taking stamina and savvy rather than actual knowledge of the vocabulary. I was hoping for a 700 for the psychological impact it would have over a nearly as impressive score in the high 600s (if anything because of the "7" in front)... similar to in marketing when prices are 1 cent less, i.e. 699.99 just so you don't choke upon seeing the 7 (but obviously this works in the opposite direction). Ok, maybe I'm not 100% totally over it. Like you though, I was aware of the grading system for the AWA and also wrote something more on the prosaic side. What a worthless test. I'm sure the adcomms will like the 720 (which is above all the average scores in every department I'm considering)... Even if I could score a little higher on a re-test, there is no way I would give ETS the pleasure of an additional $160. What a pointless waste of money that would be... /rant

Anyway, props to the OP for a great thread!

Well, brace yourself: I received a 690V! And I understood right then that the lack of a "7" would instantaneously "demote" my score, even though it's only one question or so difference. It's just psychology. So frustrating to know I was only, like, one question away. So now you can even better understand my frustration with the test. Not only am I excluded from the 99th club, but also the 700 club (yikes, not THAT club, if you know anything about American televangelism...quite happy to be excluded from that club) as well. But you scored 720V, and you are in the same *percentile* as I (despite getting several more questions right, which is actually quite difficult if you consider test conditions), even though I don't get the shiny "7," and even though my practice tests said I would score above 700. The annoying thing: at my last question, I still had three minutes left! But I couldn't go back, as I could on a paper test, and retry any tricky questions I had previously rushed due to anxieties about time (mainly, I was worried about reading comp passages cropping up near the end). I'm pretty much over it, too, and decided not to retest because I know a 690V shouldn't keep me out of the running for good programs in my particular field, and also because the thought of giving ETS any more of my money made me sick to my stomach. Another $160 dollars? Yeah, right. Anyway, I realize how this blathering might look to a humanities student who scored under 600V ("woe is me, I didn't break 700, boohoo"), and all I want to say is that you're smart and ETS is terrible and no one will really care about your score if your writing sample and SOP are great. This is just me being insecure, because I really did want a 700+, and Powerprep lied.

[Paradeplatz]

Don't overrate human readers:

- No ETS grader will spend more than two minutes on an essay

- No adcom will ever look at an essay written in 30min, even if he/she could

[gradschoolorbust!]

But you scored 720V, and you are in the same *percentile* as I (despite getting several more questions right, which is actually quite difficult if you consider test conditions)

Actually my 720V just made the 98th percentile threshold... but who's counting, lol.

We're fine! One day, when we're wizened old people, we'll look back and laugh at how ridiculous we were for caring about this, that is, if we even remember it!

May my silly dream of reaching the 99th percentile in Verbal R.I.P. This future grad student has more important things on which to concentrate (like not ending my sentences with prepositions).

As grad applicants we take the GRE SOOOOOO much more seriously than adcomms do. It's really quite obsessive. Let's just be thankful and move on with our lives!

[shepardn7]

Actually my 720V just made the 98th percentile threshold... but who's counting, lol.

We're fine! One day, when we're wizened old people, we'll look back and laugh at how ridiculous we were for caring about this, that is, if we even remember it!

May my silly dream of reaching the 99th percentile in Verbal R.I.P. This future grad student has more important things on which to concentrate (like not ending my sentences with prepositions).

As grad applicants we take the GRE SOOOOOO much more seriously than adcomms do. It's really quite obsessive. Let's just be thankful and move on with our lives!

Ahaha! I'm literally laughing now (not just "LOL"), because my assumption that you meant you received 97th per. seems to explain why I got some of those reading comp questions wrong. . .teehee. . .*cries.* Anyway, you're right about us taking it too seriously, but I think it's more an "Internet' thing than it is a general applicant thing. People on this forum especially, but elsewhere, too. It's the kind of thing that generates panicked discussion. But yes, I'm thankful it's over and I can't wait until I never have to think about it again. (!!!)

[gradschoolorbust!]

Hahaha, no worries.

I feel a little bad that we hijacked this thread with our borderline bathetic Verbal score rants.

May the AWA computer grading discussion continue...

[balderdash]

To be honest, and I don't mean to start a massive argument or to insult your writing skills (which I'm honestly certain are superior to my own), but isn't crafting your writing to the requisite format part of being a skilled essayist? I mean seriously, when you're submitting to a journal, a dissertation committee, or a conference, you're structuring your argument and changing the scope of your work to the requirements at hand. Why should the AW section be any different? Yes, I understand it's only in half an hour, and that it takes but a minute to be graded - both of which are ludicrous. But if you realize that to pander to such a system necessitates key introduction/conclusion/transition words, basic structure, clarity of prose, and impressive vocabulary, then why can't you provide them if for no other reason than to do well on the test? I would never publish something with which I wasn't comfortable, but I would certainly change my style for one essay in this situation.

Well, that's my two cents, anyway.

[UnlikelyGrad]

I would certainly change my style for one essay in this situation.

But see, the inherent problem is that you can't change your style for "one essay"--this is a timed test, and so you have to practice over and over until you can whip out the requisite style without thinking about it. If I had unlimited time to write and could consult a "cheat sheet" of tips, I would also, like you, have no problem with adjusting to the appropriate format. Scientists have to do this anyway, when submitting a paper to journals with different article guidelines.

I like to think that I'm a pretty good writer precisely because I can strike a good balance between "formulaic" and "interesting." This is an inherent part of the way I write--it takes energy for me to overcome it. Needless to say, I didn't want to invest too much effort into changing myself for a one-time test. My compromise? Before taking the GRE, I read tip sheets for the AW section, then tried to incorporate them into my essays when I was taking the test. But I didn't spend hundreds of hours trying to prep myself to get a perfect score. I ended up getting a 5: not as good a score as I thought I deserved, but not terrible, either.

P.S. I fixed the title for the OP--apostrophe errors bug the heck out of me.

[Eigen]

I think (as mentioned above), way too many prospective grad students put too much stock into GRE scores- they're a hoop you jump through, but not a particularly important one.

Talking to our AdComm, scores basically fall into three categories: Didn't meet the minimum requirement, met the minimum requirement, or the "it's really high" category. These all differ from school to school, but really all that matters is meeting the minimum for the department/school. Past that, they care more about your GPA, your research experience, and your letters of rec.

They realize that the GRE is a short, segmented window to show your skills. They realize that not everyone tests well. They know that essays were only written in 30 minutes. And while it's good to show that you can perform under pressure to meet specific, and convoluted goals, they're not going to base more on that 4 hour test than on the 4-5 years of work that you're showing in the other parts of your application.

Just something to keep in mind.

[HopefulGrad2B]

P.S. I fixed the title for the OP--apostrophe errors bug the heck out of me.

THANK YOU!! Haha, it was bugging me too!

[warpspeed]

They aren't all "egraders" or "computer graders," my best friend's mom grades AWAs for ETS. She's been a Rutgers composition professor for YEARS. And from what she's told me, every essay gets a once over with human eyes. I know its a frustrating test, but this really seems like you're trying default the possibility of you getting any score under perfect as "the computer's fault" or "ETS's fault," when in reality, it's not. This is the game, play it their way or don't play it all.

Furthermore, it irritates me that people think that just because they have high verbal scores means they got cheated on the AWA. Writing and verbal test very different things. I have a perfect score on the writing and a slightly above average (~80% percentile) verbal. Good writers are able to adapt to the needs of their work. A need here is to adapt to their dry style. If you can't change your writing style on the fly to a very basic, run of the mill standard, then really how good fo a writer are you?

Edited November 30, 2010 by warpspeed

[history_PhD]

Another verbal/writing discrepancy person here-- scored a 730 V but a miserable 4.0 on the AW. 99th and 43rd percentiles, respectively. Oh well, the verbal means more than the writing, from what I have read, so cheer up! The test is over!

[Golden Monkey]

I think (as mentioned above), way too many prospective grad students put too much stock into GRE scores- they're a hoop you jump through, but not a particularly important one.

Talking to our AdComm, scores basically fall into three categories: Didn't meet the minimum requirement, met the minimum requirement, or the "it's really high" category. These all differ from school to school, but really all that matters is meeting the minimum for the department/school. Past that, they care more about your GPA, your research experience, and your letters of rec.

They realize that the GRE is a short, segmented window to show your skills. They realize that not everyone tests well. They know that essays were only written in 30 minutes. And while it's good to show that you can perform under pressure to meet specific, and convoluted goals, they're not going to base more on that 4 hour test than on the 4-5 years of work that you're showing in the other parts of your application.

Just something to keep in mind.

This is what I've been told too. I hope it's true. My first pick for a PhD program doesn't even require the GRE. I had to take it for the other three programs I applied to. As a horrible test-taker, I had mediocre scores on the Verbal and AW, and bombed the Quantitative, which is funny (not haha funny, but peculiar funny,) because I have an engineering degree and years of work experience as an engineer, but this doesn't mean I'm good at doing 7th grade arithmetic in the form of multiple choice problems in a timed environment. I think the rest of my application must look pretty good by comparison, as I have 15 years of work experience (I waited a really long time to go back to school,) letters of recommendation from some fairly well-known faculty, and a 4.0 from my master's program. I suppose that if any program considers my dismal GRE scores more important than the rest of my application, I didn't belong there in the first place.

I'm just hoping they don't use automated bot algorithms to read Statements of Purpose too. Anyway, it's over and done with, and I'm happy about that part, and ETS is not getting another $160 out of me.

[TheDude]

I received a great score in the AW section (5.5) and received a 560 in the verbal section. I'm inclined to take the AW score as a valid measure of my intellectual abilities. However, I take it as I do with the rest of the GRE....complete horse shit, and just a formality in the application process.

[SarahSocPsy]

My first time taking it I got a 6 AW and 520 verbal. I would have definitely preferred a high verbal score instead!

[DrFaustus666]

They aren't all "egraders" or "computer graders," my best friend's mom grades AWAs for ETS. She's been a Rutgers composition professor for YEARS. And from what she's told me, every essay gets a once over with human eyes. I know its a frustrating test, but this really seems like you're trying default the possibility of you getting any score under perfect as "the computer's fault" or "ETS's fault," when in reality, it's not. This is the game, play it their way or don't play it all.

Furthermore, it irritates me that people think that just because they have high verbal scores means they got cheated on the AWA. Writing and verbal test very different things. I have a perfect score on the writing and a slightly above average (~80% percentile) verbal. Good writers are able to adapt to the needs of their work. A need here is to adapt to their dry style. If you can't change your writing style on the fly to a very basic, run of the mill standard, then really how good fo a writer are you?

I'm glad to hear that in at least some cases the "AW folklore" is not true: that folklore says that ETS mainly hires Princeton graduate students to be the human readers.

But the AW test is still a test of one very specific kind of writing---a kind that some very good writers cannot do, period.

Here's an analogy from my own field, music. Mozart for example, routinely wrote out perfectly formed entire pieces, finished in his brain, without corrections on paper. Whereas Beethoven worked and re-worked and re-worked his sketches over years (the notebooks survived, and are an amazing insight into the creative process!). My point is, equally good writers may not be all able to write equally well when under pressure of time. And thank goodness, by most accounts at least, Admissions Committees give this point of view precedence over a 30-minute timed essay, taken under pressure.

Edited December 3, 2010 by DrFaustus666

[jfsaunders]

I took the GRE twice-- received verbal scores in the 96-98%tile, and writing scores of 4.0 (45%tile). Thank goodness programs look at SOPs and writing samples!

[TheDude]

And in the 60's Miles Davis, Coleman, Coltrane, etc made it up on the spot and it was all brilliant.

I'm just messing with you!

I'm glad to hear that in at least some cases the "AW folklore" is not true: that folklore says that ETS mainly hires Princeton graduate students to be the human readers.

But the AW test is still a test of one very specific kind of writing---a kind that some very good writers cannot do, period.

Here's an analogy from my own field, music. Mozart for example, routinely wrote out perfectly formed entire pieces, finished in his brain, without corrections on paper. Whereas Beethoven worked and re-worked and re-worked his sketches over years (the notebooks survived, and are an amazing insight into the creative process!). My point is, equally good writers may not be all able to write equally well when under pressure of time. And thank goodness, by most accounts at least, Admissions Committees give this point of view precedence over a 30-minute timed essay, taken under pressure.

These threads are all the same. If we get a good writing score we want justifications that we are great writers and that adom committees will care. If scores suck we want justifications ad committees do not care.