Fundamental flaw in GRE reading comprehension test

[canberra]

Very provocative and thoughtful article that says the reading comprehension test is flawed, leading to poor scores. The article is a bit long but definitely worth reading thru.

Why the GRE reading comprehension test is fatally flawed

Edited April 26, 2011 by canberra

[Zouzax]

this is really interesting. When I took the SATs in high school, my verbal and quantitive scores were only ten points apart. Ive always considered my math and reading skills to be on par with each other. So I was really surprised when my GRE scores came in and my verbal score was 120 points below my math score. During the test, I had a really hard time reading the passages and answering all the questions in the allotted time -- that was definitely the part that slowed me down. This article explains things really well.

[canberra]

this is really interesting. When I took the SATs in high school, my verbal and quantitive scores were only ten points apart. Ive always considered my math and reading skills to be on par with each other. So I was really surprised when my GRE scores came in and my verbal score was 120 points below my math score. During the test, I had a really hard time reading the passages and answering all the questions in the allotted time -- that was definitely the part that slowed me down. This article explains things really well.

Something similar happened with me, and I wondered why. It's not like my reading skills dropped suddenly. If anything, I had begun reading more widely than when I was in high school.. One thing I find really bizarre is that all the test-preparation advice says you should read newspapers, magazines and so on. And the irony is that, as the article shows so clearly, the reading comprehension passages are not written in the reader-friendly manner of newspapers, magazines etc. All this advice is useless, it's probably worse than useless because you get used to reading reader-friendly language, rather than the boring and convoluted text of the test passages.

This article is game-changing. Wonder what will happen if someone from ETS reads it, have half a mind to forward it to them.

[MoJingly]

This article is game-changing.

I hope so. Well done.

[repatriate]

This article is a terrible example of the clear writing the author so wishes we would all create. It's full of strange digressions (such as the salaries of bankers), and the author is using the GRE as a vehicle for a broader complaint about academic writing styles that really has nothing to do with what the GRE should test.

Whether or not we ought to write accessibly (we ought), the GRE should test the ability to read the scholarly literature that is available. In the world we live in, scholarly literature is dense and convoluted in style. In most graduate classes, instructors will expect you not only to read and understand this literature without help but also to lead discussions on that same material. Why should the entrance exam test your ability to comprehend a totally different kind of prose than you will encounter in graduate school?

Edited April 26, 2011 by repatriate

[canberra]

This article is a terrible example of the clear writing the author so wishes we would all create. It's full of strange digressions (such as the salaries of bankers), and the author is using the GRE as a vehicle for a broader complaint about academic writing styles that really has nothing to do with what the GRE should test.

Whether or not we ought to write accessibly (we ought), the GRE should test the ability to read the scholarly literature that is available. In the world we live in, scholarly literature is dense and convoluted in style. In most graduate classes, instructors will expect you not only to read and understand this literature without help but also to lead discussions on that same material. Why should the entrance exam test your ability to comprehend a totally different kind of prose than you will encounter in graduate school?

I don’t think you saw this example of GRE ‘scholarly literature’ quoted in the article, because I can’t believe you’d defend a sentence like this one:

“The increase in the numbers of married women employed outside the home in the twentieth century had less to do with the mechanization of housework and an increase in leisure time for these women than it did with their own economic necessity and with high marriage rates that shrank the available pool of single women workers, previously, in many cases, the only women employers would hire.”

Far from ‘scholarly writing’, this is the writing of a person who hasn’t attended a high school class on punctuation.

And for argument’s sake, let’s say convoluted and dense writing styles are ok. But as the author says, why can’t academic writing (including the GRE passages) make use of wordprocessing features like bold, italics, underline, bullet points, and so on? These things make life easier for the reader and are widely used now because of computers, which were not widely available in earlier times. What about headlines, which all ‘scholarly literature’ articles have?

Also you mention that “In the world we live in, scholarly literature is dense and convoluted in style”. But this doesn’t mean the world cannot and should not change. Just because earlier generations wrote densely, this doesn’t mean we should continue doing the same. In the olden days, language such as thine, thee and so on were used. We don’t use language like that anymore. Times change; language cannot remain a prisoner of the past, or a prisoner of lazy and arrogant ‘scholars’ who can’t be bothered to make their writing accessible.

[repatriate]

I agree. We ought to write accessible text. That is a separate issue from what texts the GRE should sample from. The GRE should sample from the kind of texts you will read in graduate school. Unfortunately, many scholars do write like this. You will need to be able to read such writing in graduate school, regardless of whether or not it ought to exist.

[Zouzax]

I agree. We ought to write accessible text. That is a separate issue from what texts the GRE should sample from. The GRE should sample from the kind of texts you will read in graduate school. Unfortunately, many scholars do write like this. You will need to be able to read such writing in graduate school, regardless of whether or not it ought to exist.

ok now you've passed the border of making a point to being just plain obnoxious.

Edited April 26, 2011 by Zouzax

[Eigen]

this is really interesting. When I took the SATs in high school, my verbal and quantitive scores were only ten points apart. Ive always considered my math and reading skills to be on par with each other. So I was really surprised when my GRE scores came in and my verbal score was 120 points below my math score. During the test, I had a really hard time reading the passages and answering all the questions in the allotted time -- that was definitely the part that slowed me down. This article explains things really well.

The reason the scores are far apart isn't because the verbal is too hard, in my opinion, but rather because the math section is way too easy. Perfect scores are considered the "norm" and what you need to be competitive in any of the physical sciences/engineering, and that's just absurd. It's quite different from the more reasonable scores (high 600s to 700s) that are considered competitive in liberal arts.

I didn't find the passages in the GRE very convoluted or hard to follow, but that was just my personal opinion. And I think the point is well made that the passages should be quite dense, as that is the norm in academic writing- maximizing the information with a minimum amount of text.

ok now you've passed the border of making a point to being just plain obnoxious.

No, it wasn't... It was quite funny- and a nice illustration of his point.

I am of the opinion that bold, italics and underlines for emphasis have no place in scholarly writing... Just as they haven't found a place in popular writing. The idea is that you should be able to drive home your point via word choice and sentence structure without the need to resort to such glaring techniques as bold, italics, and underlines in the body of your work.

The use of headings was also mentioned, but the GRE passages really aren't long enough to necessitate headings, in my opinion.

Edited April 27, 2011 by Eigen

[canberra]

I didn't find the passages in the GRE very convoluted or hard to follow, but that was just my personal opinion. And I think the point is well made that the passages should be quite dense, as that is the norm in academic writing- maximizing the information with a minimum amount of text.

You may find the passages easy to follow but the reality is that many people find them very difficult. This means the fault lies with the writer, not just with the readers.

Moreover, I would hardly say the following sentence maximizes information with minimum amount of text.

“The increase in the numbers of married women employed outside the home in the twentieth century had less to do with the mechanization of housework and an increase in leisure time for these women than it did with their own economic necessity and with high marriage rates that shrank the available pool of single women workers, previously, in many cases, the only women employers would hire.”

I am of the opinion that bold, italics and underlines for emphasis have no place in scholarly writing... Just as they haven't found a place in popular writing. The idea is that you should be able to drive home your point via word choice and sentence structure without the need to resort to such glaring techniques as bold, italics, and underlines in the body of your work.

The question isn't whether something has a place in scholarly writing or not. The question is, does the writing help make life easier for the reader? If bold, underline etc help the reader, why should they not be used? Who made up the rules that scholarly writing should be not include the use of wordprocessing features (bold, italics etc)? Why can't we question these rules, given that they seem to come from ancient times when scholars didn't have computers?

The only reason I can think of is insecurity. 'Scholars' don't want others to think that because their writing is easy to understand, the scholar is really not very intelligent.

The use of headings was also mentioned, but the GRE passages really aren't long enough to necessitate headings, in my opinion.

If a headline helps make life easier for the reader, why not?

[Lyra Belacqua]

From the article:

I’m no biologist and don’t know what the word ‘genus’ means - I assume it’s some kind of categorisation.

And that's really all you need to know. Knowing the definition of "genus" is actually not important to understanding the passage (whereas in a passage about cladistics or speciation or something like that, it might be). The point of the sentence that includes the word "genus" has nothing to do with what a genus is; neither does the purpose of that sentence in the paragraph. It's there to give evidence for the "relationship between fever and iron" discovered by some guy named Garibaldi (specifying what exactly it was that he discovered). The sentence could have been "He found that microbial synthesis of siderophores -- substances that bind iron -- in Salmonella declined at environmental temperatures above 37° C and stopped at 40.3° C."

So the GRE is actually doing what the author just claimed it's not doing: explaining possibly unfamiliar terms. They didn't have to let you know explicitly that Salmonella is a genus of bacteria…but they did!

And I'm not about to claim that the GRE should be in the business of testing people's general knowledge (like knowing what a genus is, or who Friedrich Engels was), but I still don't think it's an entirely bad thing if your knowledge of history, science, or literature gives you an extra clue to understanding something (e.g., "this sentence is quoting Engels, so it will probably be about social class and/or economics").

I'll grant him one thing, though: the issue of people taking the GRE who don't understand US cultural references (or, for that matter, aren't native speakers of English) is indeed quite problematic, and I don't know what should be done about that.

I was reading about the history of the GRE a while ago, and I thought it was awesome that they used to have the "analytical section" (although I was disappointed that they got rid of it). Apparently, the GRE used to have LSAT-style logic games – how cool is that? (And, yeah, I wonder why it was taken out. God forbid grad students should be expected to perform tasks of logical reasoning, eh? )

[canberra]

Knowing the definition of "genus" is actually not important to understanding the passage (whereas in a passage about cladistics or speciation or something like that, it might be). The point of the sentence that includes the word "genus" has nothing to do with what a genus is;

Why use the word 'genus' at all then? There seems to be absolutely no editing that happens with these passages.

The debate apart, the premise of the 'flawed GRE' article is that the reader's ability to comprehend partly depends on the quality of writing.

I really, really, really don't see anything wrong with that.

[Lyra Belacqua]

They think you might never have heard of Salmonella, so they are telling you what it is: a genus of bacteria.

Although…most people learned "King Philip Came Over For Greasy Spaghetti" in middle school anyway, so I don't really see what the problem is. Well, if you've ever been told as a kid why you shouldn't eat raw cookie dough, then you know what Salmonella is too, so I don't know why they felt like they had to explain that either.

If they had decided not to use any words they thought people might not know, the passage probably would have come out sounding a little like "Once upon a time, there was a dude who did some research on beings that can't be seen without a microscope and found out that there was a link between temperature and iron levels."

[canberra]

"Once upon a time, there was a dude who did some research on beings that can't be seen without a microscope and found out that there was a link between temperature and iron levels."

Your line actually explains the point of the passage far better, and enables the reader to comprehend far more quickly! The explanation of Salmonella etc can follow, but in essence, your sentence is really what the passage is about. And your use of "once upon a time" means that it's definitely not boring.

It's not 'scholarly writing' but it gets the point across in a fun way. What's the problem?

I would bet that far more people would understand your line in a few seconds, compared to muddling through the GRE passage wondering what on earth the main point is.

[Ergative Avenger]

The author of the article has a point with regard to cultural bias, but the long-winded crowing about "difficult" passages lost me entirely. I'm sorry, but I just did not have any problem with the passages in the article or in the test. I got a very good score on the GRE Verbal, but I do not consider myself to have some kind of unusually strong reading comprehension ability. I am frequently bogged down by literature in my own field.

The author suggests that reading passages come with headings that explain the main idea. Sure, let's do it. But then we can't have questions like "What is the title of this passage?" or at least, few of us on this board will ever run into a question that easy due to the adaptive nature of the test. A question like that does nothing to discriminate between students with poor reading comprehension and those with good reading comprehension. Instead, the test will have to ask about the main idea of untitled sub-passages, which ends up with the same "problem" identified by the author, of test questions being difficult and hurting his grad student friends' feelings or whatever. It's also important to note that in scholarly literature, one will often have to understand the main idea of paragraphs within a text, and not every individual paragraph will come with a helpful title. At a certain point, as a scholar, you have to supply some of your own brain power.

I did have one beef with the reading comprehension portion, but it had nothing to do with the texts themselves. From doing many practice tests and reading answer explanations, I noticed that incorrect answers were frequently described as containing misrepresentations or oversimplifications of the author's intent. In practice, I often found that all five answer choices were guilty of some degree of misrepresentation or oversimplification, leaving me to guess which one the test-maker wanted. Usually, this was not too difficult.

“The increase in the numbers of married women employed outside the home in the twentieth century had less to do with the mechanization of housework and an increase in leisure time for these women than it did with their own economic necessity and with high marriage rates that shrank the available pool of single women workers, previously, in many cases, the only women employers would hire.”

For example, I don't see any grammatical problem with this sentence or, for that matter, any major stylistic problem. All conjoined and compared phrases are in perfect parallel. I guess the problem is supposed to be with the final appositional phrase with two fronted adverbials. The first comma could be replaced with a dash, or "in many cases" could be moved to the end, but otherwise I don't see how this sentence is difficult to understand. It's long. So what? It's a concise and precise statement of the complex relationships between factors bearing on married women's employment. Scholarly writing, by necessity, frequently deals with complex relationships. Writing in simplistic terms as if for a high school audience leads to an oversimplification of the relationships being discussed. That's fine for high schoolers. It's not appropriate for writing aimed at grad students and scholars working in that field, who require detail and accuracy.

For that matter, the level of difficulty of a passage is largely irrelevant to a computer adaptive test that reports percentile scores, like the GRE. The intent is to identify a student's relative ability to comprehend scholarly writing, as compared to the total body of test-takers. As long as the test stratifies test-takers into those that do well and those that don't (which means, ideally, less than 1% of test-takers should receive the maximum score), absolute difficulty is irrelevant. If the test is simply made easier, somehow, then everyone will just get higher scores. Well yay, higher test scores are always good, right? At best, the percentile rankings will be unchanged. At worst, the verbal section becomes like the too-easy quantitative section, where it is impossible to score in the 99th percentile, and the test is therefore unable to identify the top 1% of students in that section.

To be sure, there are flaws in the GRE. Hopefully, the new test will address some of these, but this article hardly identifies any real flaws. The GRE reading comprehension portion tests the ability to read and understand scholarly literature. All scholarly literature is in someone's subfield. By sheer mathematics, it's unlikely to be in your subfield. (I didn't get any readings in my field, broadly speaking.) Likewise, every text is written from someone's cultural context. There are no culturally neutral texts. I wouldn't have any complaints about reading a text on Indian music, and in fact, many texts drawn from the humanities deal with cultures and historical periods that are likely to be unfamiliar to American test-takers. I would advocate there being more such texts. However, no matter what your culture or academic specialty, the odds are overwhelming that the subject matter is going to be unfamiliar to you. That's part of what makes the GRE a test, rather than a four hour self-esteem booster.

[Lyra Belacqua]

The author of the article has a point with regard to cultural bias, but the long-winded crowing about "difficult" passages lost me entirely. I'm sorry, but I just did not have any problem with the passages in the article or in the test. I got a very good score on the GRE Verbal, but I do not consider myself to have some kind of unusually strong reading comprehension ability. I am frequently bogged down by literature in my own field.

Same here. I'll admit that I sometimes have trouble reading linguistics papers that have a lot of mathematical formulas, and that's something I'm going to have to work on when I get to grad school. But luckily for me, the GRE didn't test my ability to read stuff like that! XD

Your line actually explains the point of the passage far better, and enables the reader to comprehend far more quickly! The explanation of Salmonella etc can follow, but in essence, your sentence is really what the passage is about. And your use of "once upon a time" means that it's definitely not boring.

It's not 'scholarly writing' but it gets the point across in a fun way. What's the problem?

I would bet that far more people would understand your line in a few seconds, compared to muddling through the GRE passage wondering what on earth the main point is.

I wouldn't have thought it was that hard to figure out that my sentence and the Garibaldi sentence mean pretty much the same thing.

But even though the author of the article apparently isn't American, I lost my sympathy for whatever cultural disadvantage he might have had when he gave Friedrich Engels as an example of a historical figure that it's unreasonable for the GRE to assume that people will know. I mean, the Communist Manifesto is so internationally prominent that I do think it would be safe to assume. And it's not like those guys were from Anglophone lands, either!

Edited April 29, 2011 by Lyra Belacqua

[canberra]

I noticed that incorrect answers were frequently described as containing misrepresentations or oversimplifications of the author's intent. In practice, I often found that all five answer choices were guilty of some degree of misrepresentation or oversimplification, leaving me to guess which one the test-maker wanted. Usually, this was not too difficult.

Well, if all the answer choices were guilty of some degree of misrepresentation or oversimplification, obviously even the correct answer was badly written too. The writing should be should be so clear that you should not have to guess. Further you should be thinking about what the author of a passage wants you to understand, rather than what the test-maker thinks you should understand . If there's a difference between what the author says, and what the test-maker thinks the author is saying, then clearly there''s a problem. Unless the author is the one vetting his own passage questions/answers when the test is being created, we have no way of knowing whether the test-maker is in synch with what the author is saying in the passage.

For that matter, the level of difficulty of a passage is largely irrelevant to a computer adaptive test that reports percentile scores, like the GRE. The intent is to identify a student's relative ability to comprehend scholarly writing, as compared to the total body of test-takers. As long as the test stratifies test-takers into those that do well and those that don't (which means, ideally, less than 1% of test-takers should receive the maximum score), absolute difficulty is irrelevant. If the test is simply made easier, somehow, then everyone will just get higher scores. Well yay, higher test scores are always good, right?

That's the problem with percentile based tests. To make the test supremely difficult, you could just have one long and complex passage comprising a single sentence, containing multiple strands of thought with no punctuation marks. There certainly will be a few people who will succeed in this test. Does this mean everybody else can't read and comprehend as well?

The aim of scholars should be for them to have their ideas and theories contested and debated. There should be no confusion about what they're saying but the reality is that there is confusion. Instead of arguing about the ideas, in the passages we argue about "did the scholar mean this, or did the scholar mean that?"

[repatriate]

As long as the test stratifies test-takers into those that do well and those that don't (which means, ideally, less than 1% of test-takers should receive the maximum score), absolute difficulty is irrelevant.

I agree with most of what you said, but this is incorrect. In order to closely approximate a test-taker's "true" score, you need questions at many levels of difficulty. Each question is associated with an approximate level of ability at or above which a test-taker will answer correctly and below which a test-taker will answer incorrectly. You must provide questions at many levels in order to come up with a good guess of someone's actual level of ability. This is why the computer adaptive nature of the GRE is important: once the computer program figures out that you are above or below a certain ability level, it starts serving you questions in the range it thinks you are so that it can discriminate more finely among people close in "true" score.

So, let's say abilities range from 1 to 100, and people with a true ability of 50 or more will generally get a certain question right, and people with a true ability of less than 50 will get the question wrong. Then, this question only divides people into two bins: at-or-above-50 and below-50. By including questions at all different ability levels (and preferably a couple at each), you make those bins smaller and smaller.

So both easy and hard questions are necessary for a standardized test like the GRE.

[spctle342]

Let me first qualify this by saying I have not yet taken the GRE. To a degree, I understand the point the author of this article is trying to make, but I don’t share the sentiments. Scholarly literature isn’t intended to be palatable for general audiences. While some would argue that it should be simplified to promote some sort of free access to all, I think that just invites even more bastardization of science, and I can’t really get behind that.

If I’m reading a psychological journal, I don’t expect to see the same crap I’d come across in Psychology Today, complete with bulleted lists, colorful graphics, a million headlines, and some tangentially relevant photograph of a nearly nude woman wearing oven mitts. I expect to have to put forth the effort to interpret the research. Quite honestly, I find it condescending that an author should feel compelled to do this work for me. Reading comprehension is just that—readers must employ their own techniques to sort through the information and determine what’s being said.

Admittedly, I’ve ceremoniously burned a textbook or two that I felt was needlessly verbose, complicated, or otherwise stylistically worthless. As an undergraduate student, I expect to encounter textbooks that are more user-friendly than those I would come across during graduate study. At the same time, I feel embarrassed for myself and the authors when I open up a new textbook to find comics, jokes, and the definitions to every basic term hanging around the margins of every page. It’s insulting to my intelligence as a college student.

Ultimately, I’m not convinced that this article was so much an informative criticism of the GRE as it was a “how-to” in writing for audiences with clinically-deficient attention spans. Though I must note I couldn’t get past the first three pages…because I got bored.

[canberra]

I expect to have to put forth the effort to interpret the research.

You're absolutely right. You should be putting in effort to interpret the research, not the scholar’s writing.

[Milo_10011]

Sorry, but the article cited is just a terrible piece of writing.

The point (if there is one) of the GRE is that it is designed to be confusing. I managed to slog through about three pages of the article before reaching my "okay, the author isn't keeping my attention" point, and in that time, the author did not get to the only legitimate criticize about the GRE there is:

Often, the difference between the "almost-right" answer and the "exactly-right" answer is so arbitrarily _____ that there is no legitimate way to select.

A. tiny.

B. minuscule.

C. small.

D. enormous.

The correct answer, of course is B. Why not A or C? Minuscule indicates a smaller degree of difference than small or tiny. (Small would be a $1 error in your paycheck. Tiny would be a 10 cent error. Minuscule would be a fraction of a penny.)

The differences involved are, of course, false. Why? No one, even (or especially) in academia, reading a paper, is going to make a distinction between A B and C. Why not? Because the differences between those words is so trivial, and has been so smudged by common usage, that going into a paper with such a pedantic approach will put you at a disadvantage.

The passage the author uses about iron and fever in warm-blooded animals? It actually disproves the point he (she?) is trying to make. Much of academics is running into information you are NOT familiar with. (And it's sad that the author admits to not understanding what a genus is.) The passage's fundamental content is clearly understood through reading for context.

Also, a lot of academics is about reading VERY, VERY, VERY dry material. It isn't putting your feet up, drinking a beer and reading Stephen King's latest door-stop 450-page short story.

When I was an undergrad, one of the chem teachers told me something that has stuck with me for decades. "Milo, how many people are in the lecture? About 300? And that's just ONE lecture. We have these lectures four times a day, five days a week. That's 6,000 students taking Chem 101. And we're just one school. Do you think that many chemists are needed? Every year? Of course not. Chemistry is hard, and a lot of it is boring unless you really are genuinely interested in the subject. The worst thing we could do to these students is to tease them along until their first semester as juniors, giving them easy course and then turning up the heat when it's too late for them to change majors. So we give them an intro course and deliberately make it hell. The only ones who'll stick it out are the ones who'll end up continuing in chemistry later on."

If nothing else, that is the GRE's single use: to chase away the students who simply can't cut it.

[canberra]

Also, a lot of academics is about reading VERY, VERY, VERY dry material. It isn't putting your feet up, drinking a beer and reading Stephen King's latest door-stop 450-page short story.

There's no law that says academic reading material has to be dry. Universities are places of learning, where we explore new and exciting things. It's the faculty who don't feel the excitement, who write in a dry manner.

[Lyra Belacqua]

There's no law that says academic reading material has to be dry. Universities are places of learning, where we explore new and exciting things. It's the faculty who don't feel the excitement, who write in a dry manner.

They obviously DO feel excitement – otherwise they wouldn't be doing research in their field of choice, would they? But they have to use "boring" and "dry" things like technical terms and precise definitions, because that's the way stuff works.

[canberra]

But they have to use "boring" and "dry" things like technical terms and precise definitions

The sentence below from the GRE sample passage is not a definition, but I would hardly say it is written with any degree of precision.

The increase in the numbers of married women employed outside the home in the twentieth century

had less to do with the mechanization of housework and an increase in leisure time for these women

than it did with their own economic necessity and with high marriage rates that shrank the available

pool of single women workers, previously, in many cases, the only women employers would hire.

[spctle342]

The sentence below from the GRE sample passage is not a definition, but I would hardly say it is written with any degree of precision.

The increase in the numbers of married women employed outside the home in the twentieth century

had less to do with the mechanization of housework and an increase in leisure time for these women

than it did with their own economic necessity and with high marriage rates that shrank the available

pool of single women workers, previously, in many cases, the only women employers would hire.

There is nothing wrong with that passage. In my opinion, it reads very well. Do the commas confuse you? I suppose the last few words could have said something to the effect of "in many cases, the only women employers would previously hire." What is your criticism here? How would you have written the passage?