GRE Twisting Experimental Sections Based On Intended Degrees

[TakeruK]

44 minutes ago, skhann said:

Extra AW? I would have loved that. Did you appear recently? I've not heard many getting experimental AWs.

No, this was way back in 2011, when it was the old format (i.e. not the "Revised General GRE" which started Aug 1, 2011).

44 minutes ago, skhann said:

This is subjective. Non-math ones dread Quant and the STEM types get scared of verbal.

I actually would have preferred an extra verbal section over an extra Quant section. For me, the Verbal questions were "simple" in the sense that I either knew the answer or I didn't know the answer. There was not much extra anxiety over whether I was approaching it the right way or not. I prepared for this section via brute force just memorizing as many words as I could. For the Quant section, I didn't really prepare very much since I already had a lot of training in math. But that's where I start to second guess myself. Did I make a simple mistake? Am I overthinking the problem? Is this actually the right approach? etc. A lot more stressful. But as you say, it's all subjective and depends on each person!

[spamhaus]

1 hour ago, TakeruK said:

For what it's worth, I also put my intended major to be a STEM one and my experimental section was a quantitative one. Maybe they just put more of those in, or maybe the really small sample size we have here isn't enough to draw any conclusions!

I'm guessing you're right regarding the small sample size, even when a single individual takes the test multiple times the results are simply anecdotal. Though, the way I look at it is this: how much better would my quantitative score be if I was only given two quantitative sections instead of three? That is something that ETS can easily determine, as there are likely many people who take the GRE multiple times, so they likely have the data to do such an analysis.

Unfortunately, I have not seen that reported in any of their literature. I've only seen one fact regarding multi-test takers: they generally do better the second time they take the test versus the first time. If it is indeed a 50/50 toss-up as to getting an experimental quantitative versus verbal section, that could theoretically provide a partial explanation for the discrepancy assuming: 1) people who get an experimental section that coincides well with their intended subscore goals are less likely to retake the test, 2) people who get an experimental section that negatively affects their intended subscore goals are more likely to retake the exam. That is a gross over simplification. For example, it excludes the additional familiarity with the exam that people gain by taking it a second time.

Even ignoring the simplification, this is simply wild conjecture! Though, running through scenarios where my notion of the ideal test setup works out in my favor gives me solace at times, i.e. it sometimes feels nice to whine and complain :-D.

[skhann]

UPDATE: Just back after re-taking the test (I'm an international applicant) and got three Quant sections  

For me, this appears to be a set pattern. Obviously this is my experience and also of some other respondents.

[Cal1gula]

Interesting thread! FWIW, I chose a STEM major and ended up with an extra verbal section.

Edited January 3, 2018 by Cal1gula

[Bayesian1701]

On 12/29/2017 at 11:54 AM, skhann said:

From the responses here and my personal experience, it can be safe to assume there is an 80 percent chance that social sciences/humanities students will get a quant experimental section. Super bad but super possible.

1

We are no are near close to deciding that is the case.   So far we have anecdotes.  There is no real reason outside to assume this is the case outside of anecdotes.  I am sure you could get a real small p-value on this, but that is beside the point.   Even if there is a probability of 1% of this happening based on the data there are million of GRE takers and this "pattern" I am sure would occur in dozens of the small samples. 

I hate ETS but I really doubt this actually what is happening.   On the experimental section to have ANY validity as research it would have to represent the population of test takers.  They can do this two ways make it totally random or possibly stratify so half of the people within a field would get a quant.   This is an unrepresentative group of responses we have had so far in this thread, and your friends probably don't represent the GRE test takers across the globe.  And if this were real I feel like it would have been discovered by now outside of a gradcafe thread.  

I think this falls into the pattern of humans looking for a pattern that is not there.   @skhann may have seen a "trend"  among a few people, and then people "confirmed" his(her)  belief on this thread.  

I am not trying to be mean or condensing here but the reports in the thread mean nothing.   This genuinely scares me as a statistician and I hope that you do not continue making claims and generalizations based on small biased samples in your research.   Suggesting that this is to make ETS more money is a conspiracy theory.  This is exactly how irreproducible science happens.   Please please please please please take a statistics class taught by a statistician (with a statistics degree) in graduate school and ask a statistician for help in statistics if you need it in your research (I think you do).   A great deal of academic applied statisticians spend their entire careers collaborating and consulting with other researchers and generally love helping people make sure their experiments have the best methodology and data analysis possible.  Please read this which is by a prominent statistician which includes articles on the replication crisis (there are more in the comments)  that are accessible to people without a strong math background.  I do research in statistics for social science and I see things like this far too often.   I am not doubting your skills or abilities in your field.  This is my area and I feel like I have an ethical obligation to say something.   

For the record, I got three quant sections, and I actually would have loved a third verbal section.  

[cabbysaurus]

Interesting thread! I agree with Bayesian above that it should be (and hopefully is) random though. 

I only took the GRE once (thankfully) and got three verbal sections. I would have much rather had three quant sections though. The verbal sections really exhausted me because I had to stare at the computer so much. The quant problems, on the other hand, involved more work using paper and pencil so I felt that my eyes got a break.

One thing that was frustrating for me was that all of the practice exams I took, even the ones from ETS, only had four sections of verbal and quant. Adding in a fifth section on test day was a lot worse than I expected.

[skhann]

On 13/01/2018 at 3:55 AM, Bayesian1701 said:

We are no are near close to deciding that is the case.   So far we have anecdotes.  There is no real reason outside to assume this is the case outside of anecdotes.  I am sure you could get a real small p-value on this, but that is beside the point.   Even if there is a probability of 1% of this happening based on the data there are million of GRE takers and this "pattern" I am sure would occur in dozens of the small samples. 

I hate ETS but I really doubt this actually what is happening.   On the experimental section to have ANY validity as research it would have to represent the population of test takers.  They can do this two ways make it totally random or possibly stratify so half of the people within a field would get a quant.   This is an unrepresentative group of responses we have had so far in this thread, and your friends probably don't represent the GRE test takers across the globe.  And if this were real I feel like it would have been discovered by now outside of a gradcafe thread.  

I think this falls into the pattern of humans looking for a pattern that is not there.   @skhann may have seen a "trend"  among a few people, and then people "confirmed" his(her)  belief on this thread.  

I am not trying to be mean or condensing here but the reports in the thread mean nothing.   This genuinely scares me as a statistician and I hope that you do not continue making claims and generalizations based on small biased samples in your research.   Suggesting that this is to make ETS more money is a conspiracy theory.  This is exactly how irreproducible science happens.   Please please please please please take a statistics class taught by a statistician (with a statistics degree) in graduate school and ask a statistician for help in statistics if you need it in your research (I think you do).   A great deal of academic applied statisticians spend their entire careers collaborating and consulting with other researchers and generally love helping people make sure their experiments have the best methodology and data analysis possible.  Please read this which is by a prominent statistician which includes articles on the replication crisis (there are more in the comments)  that are accessible to people without a strong math background.  I do research in statistics for social science and I see things like this far too often.   I am not doubting your skills or abilities in your field.  This is my area and I feel like I have an ethical obligation to say something.   

For the record, I got three quant sections, and I actually would have loved a third verbal section.  

I guess you're overthinking it. I never claimed to be conducting a statistical study in an academic setting. I was very clear about sharing my own experiences and those I know and others on this forum shared theirs. Thanks for your input and for the link though. Appreciate it.

[skhann]

On 13/01/2018 at 4:28 AM, cabbysaurus said:

Interesting thread! I agree with Bayesian above that it should be (and hopefully is) random though. 

I only took the GRE once (thankfully) and got three verbal sections. I would have much rather had three quant sections though. The verbal sections really exhausted me because I had to stare at the computer so much. The quant problems, on the other hand, involved more work using paper and pencil so I felt that my eyes got a break.

One thing that was frustrating for me was that all of the practice exams I took, even the ones from ETS, only had four sections of verbal and quant. Adding in a fifth section on test day was a lot worse than I expected.

Obviously this was only an observation, confirmed by some other respondents. P.S. your quant background and three verbal sections add to the statistic. Just saying

[TakeruK]

@skhann I do agree with everything @Bayesian1701 said. This thread is a good place to vent about frustrations or share experiences. But we should not interpret it to mean anything or accuse ETS of anything. I've read through everything here and remain 100% unconvinced there's anything weird going on with experimental sections!

But I do think that the point that getting an extra section in an area you're weaker in can cause test fatigue to be unfairly applied! So my solution would be to have an extra section for both Verbal and Quant! That would be even more test fatigue but at least it's more fair? The even better solution would be to pay people to do these sections though.

[skhann]

@TakeruK You're entitled to your opinion. There's no need for experimental sections IMO. Will take the test shorter and manageable for a large number of takers, irrespective of the nature of the experimental section. Having two experimental sections will be a big no-no though.

[TakeruK]

26 minutes ago, skhann said:

@TakeruK You're entitled to your opinion. There's no need for experimental sections IMO. Will take the test shorter and manageable for a large number of takers, irrespective of the nature of the experimental section. Having two experimental sections will be a big no-no though.

I'm not sure what you are referring to. If it's the statistical rigor of the argument here, then it's not an opinion, it's a fact. Nothing in this thread provides any evidence at all of a correlation between intended major and the experimental section. I won't pursue the topic further if you don't want to discuss it, but in a world where people misunderstand and misinterpret data all the time, stating something is not statistically significant when the data does not support it is not just an opinion. That said, I'm not going to force anyone into a discussion about stats, so that's all I'll say unless you want to discuss it further.

As for the necessity of experimental sections, I agree with you that in my ideal world, there would be no experimental sections for test takers. (well I would prefer the GRE doesn't exist at all!) But if the GRE does exist, there is definitely a need to test new questions---it would be totally unfair to score applicants without any data on how difficult a question is. But since ETS is a business, testing should be on their own expense and they should be finding other means to test questions without extra burden on test takers (that already pay a ton of money to do them!!). 

I was not really serious about the two experimental sections, it was just a little cynical "humour" but sorry if that didn't translate to text properly.

[spamhaus]

43 minutes ago, skhann said:

There's no need for experimental sections IMO.

The experimental sections are needed so future test questions are based on the results of a representative sample of test takers. That's how they gauge how difficult the questions are in order to give an adaptive test. Remember your first two sections (one verbal, one quant) are middle of the road in terms of difficulty, then they decide whether you should get a more/less difficult second section. They have to base that on real test metrics, which is what the data from the experimental sections provide.

12 minutes ago, TakeruK said:

But since ETS is a business, testing should be on their own expense and they should be finding other means to test questions without extra burden on test takers (that already pay a ton of money to do them!!).

I have a feeling that it's actually more accurate to include the experimental section in an actual test than it would be to pay individuals to take tests. I have a strong suspicion that there would be selection bias with the approach you propose.

Edited January 14, 2018 by spamhaus

[skhann]

@TakeruK Paying people to test questions looks like a good idea, especially for the test takers who have to sit for another 30-35 minutes in what might as well be an additional section not based on their expertise/background. Still, as @spamhaus mentioned, it does come with its own set of limitations and biases.

 @spamhaus Your point brings forth the actual need why there is adaptive testing and exp sections in the first place. To me this just looks like a ploy to ensure a second attempt by a large number of takers. Obviously this is my observation only. Some people are bad at standardized testing but do well academically. Others are great in testing but are average performers in academic settings. GRE can be used to get a basic benchmark like scoring 150 in each section as a minimum cutoff, for example. We all know in reality many programs aim for the sky in test scores and don't even look at other application materials if the scores are below 320/330 etc.

[spamhaus]

7 minutes ago, skhann said:

 @spamhaus Your point brings forth the actual need why there is adaptive testing and exp sections in the first place. To me this just looks like a ploy to ensure a second attempt by a large number of takers. Obviously this is my observation only.

I believe in order to get a more nuanced assessment of individuals' abilities the standardized tests that ETS administers have been changed to be adaptive. There is a very large variability in the test taking populace: some individuals have little to no college level math, while others are in extremely math intensive subjects. In order to be able to differentiate people's abilities they need to provide more test questions. One way to effectively provide a large number of test questions while not making the test longer is to base it on an adaptive measure. Questions that are either too difficult or too easy for a given test taker, based on some baseline, can simply be skipped. This is most demonstrable on the quantitative section of the new "revised GRE". This link contains a table comparing the old GRE with the new GRE in terms of scores. In the old GRE the highest quantitative score was 800, which was the 91st percentile, i.e. people who received this score did better than 91% of all test takers. That is only equivalent to a 166 in the new GRE. By making the test adaptive they were better able to differentiate people, despite the fact that the new scoring has 41 possible scores, while the old scoring system had 61 possible scores. Here's a table breaking down test takers' scores for the new GRE.

 

30 minutes ago, skhann said:

We all know in reality many programs aim for the sky in test scores and don't even look at other application materials if the scores are below 320/330 etc.

It's hard to know and depends highly on the school and program. Some schools value the GRE very highly, others care for it less so. Considering that I've never been on an admissions committee, feel free to take that with a healthy dose of skepticism. That said, a counter-example of expecting sky-high GRE scores is MIT. In fact they do not require the GRE for admissions (at least not in my program EECS). There is no place to provide that information in their application. I would have LOVED the opportunity to provide that information, because I graduated from undergrad over a decade ago, and was not very mature at the time, so my GPA is lower than what I am capable of. Having the GRE being able to "make-up" for my low GPA would have been a nice addition to my application.

[skhann]

@spamhaus Thanks for the detailed response. GRE scores do play a key role in social sciences where everyone including MIT asks for the sky.

[ltr317]

I got three verbal sections and I applied to history programs.  Like @cabbysaurus, the additional verbal section really made me tired from staring at the computer screen.  After the AWA sections my test started with the first quant. section, then the first verbal and after the break, the second verbal, followed by the second quant. and finally with the third verbal.  I found the second and third verbal sections difficult after completing the first, which I found fairly easy.  I ended up scoring fairly well on the verbal, but could have done even better if I didn't have eye fatigue with the second and third verbal sections.  My ETS diagnostics confirmed my suspicions as I had three wrong answers for the first verbal and eleven wrong on the second.  I don't know whether the second or third was the experimental section, but because of fatigue, I lost concentration and was guessing by the 14th or so question on either section.   

[skhann]

Thanks for your input @ltr317 Yes the fatigue part is there but I guess the experience is very subjective. For me, bungling on quant sections stressed me out and getting an experimental quant in both attempts was like adding fuel to fire. I would rather have had a verbal exp both times. P.S. If you don't mind my asking, can you share your verbal score? I just want to know what's the final impact of having 17 correct answers in the first section with 9 correct ones in the second.

[ltr317]

@skhann we definitely had different experiences.  I wasn't expecting to do well on the quant. sections since math is not my strong suit and was expecting to score no more than 20 percent.   But after struggling with the first section, I took a devil may care attitude with the second section, and ended up with a total score slightly above the mean.  The point is that I was able to give my eyes some rest, unlike the verbal sections.  Anyway, I wanted to post here that I had three verbal sections even though I  indicated that I was applying to history programs at the beginning of the test.  So the small sample size on this thread is no indication of what ETS is actually doing.  

 

p.s. I'll pm you.  

[TakeruK]

3 hours ago, spamhaus said:

I have a feeling that it's actually more accurate to include the experimental section in an actual test than it would be to pay individuals to take tests. I have a strong suspicion that there would be selection bias with the approach you propose.

Agreed. I just think the downsides of bias in these paid test takers is better than the downsides of adding to test fatigue. The paid test takers may not represent the whole pool of actual test takers, but you can get the paid test takers to also answer current GRE questions so then you can calibrate the paid group with the actual group. I'd imagine that it might be hard to properly calibrate the highest and lowest scoring actual test takers (as the paid group may not represent these populations). But ETS should bear the responsibility and hardship of keeping up their own test metrics. One way they can recruit is to invite actual GRE test takers to do an experimental test, for free, after they complete a regular test. They can invite 1 in every 10 test takers or whatever and since they know the recruit's test score, they can invite people so that the population of paid test takers match the actual test taker population (in score at least, but there are other factors too).

Beyond that, I believe there are other things ETS can do in order to get the data it needs without involving actual test takers. They have tons of test data that eventually they should be able to train an algorithm to accurately predict how the population of test takers will respond to a particular question. 

[skhann]

20 hours ago, TakeruK said:

Agreed. I just think the downsides of bias in these paid test takers is better than the downsides of adding to test fatigue. The paid test takers may not represent the whole pool of actual test takers, but you can get the paid test takers to also answer current GRE questions so then you can calibrate the paid group with the actual group. I'd imagine that it might be hard to properly calibrate the highest and lowest scoring actual test takers (as the paid group may not represent these populations). But ETS should bear the responsibility and hardship of keeping up their own test metrics. One way they can recruit is to invite actual GRE test takers to do an experimental test, for free, after they complete a regular test. They can invite 1 in every 10 test takers or whatever and since they know the recruit's test score, they can invite people so that the population of paid test takers match the actual test taker population (in score at least, but there are other factors too).

Beyond that, I believe there are other things ETS can do in order to get the data it needs without involving actual test takers. They have tons of test data that eventually they should be able to train an algorithm to accurately predict how the population of test takers will respond to a particular question.

I really liked this training the algorithm part. ETS has literally tested hundreds of thousands of GRE takers. I'm sure it won't be too difficult for the geeks there to tweak the algorithm regarding adaptability without taxing the actual test takers both physically and economically.

Edited January 15, 2018 by skhann

[senorbrightside]

Spanish/Journalism major applying for a master in Spanish literature chiming in. I was given three quant sections. I cursed. As an aside, none of my programs look at the scores but require it because the grad school requires it. I did pretty decent on it all things considered. Definitely an interesting hypothesis. 

[spamhaus]

I came across a really salient NYT opinion piece regarding probabilities and how people seem to interpret them.

"But I’ve come to realize that I was wrong about a major aspect of probabilities.

They are inherently hard to grasp. That’s especially true for an individual event, like a war or election (*or GRE test). People understand that if they roll a die 100 times, they will get some 1’s. But when they see a probability for one event, they tend to think: Is this going to happen or not?

They then effectively round to 0 or to 100 percent."

* I thought of this thread when reading the article.

[Len]

I'm a social sciences major and got an extra Verbal section both times that I took the test.

I don't think we can draw any reliable conclusions from a handful of anecdotal cases... ETS states that the experimental section is random, and we have no reason not to believe that is true. 

[LookingforMM]

On 1/16/2018 at 8:52 AM, Len said:

I'm a social sciences major and got an extra Verbal section both times that I took the test.

I don't think we can draw any reliable conclusions from a handful of anecdotal cases... ETS states that the experimental section is random, and we have no reason not to believe that is true. 

Especially as we have several people here, such as yourself, whose experience goes against the original hypothesis.  How prevalent something "feels" to a given individual and how prevalent it actually is are often completely different.