NSF GRFP 2011-2012

[deepbreath]

Looking at other people's reviews, this whole process seems even more arbitrary. I got E/E VG/VG VG/E which resulted in an HM. None of the comments said anything even remotely negative. I see other people in my discipline with equivlaent or even worse responses. I guess things like high school location, gender, and difficulty of reviewer really matters a lot.

[bgodbgg]

I agree that it seems very arbitrary. I got a G/VG, G/G, and VG/VG and I got one. (Computer Science - other, bioinformatics) I'm guessing that the other category of bioinformatics helped and my broader impacts is really good although my academics aren't the strongest (mediocre GPA). I had heard from past people who got it that the broader impacts was almost more important than the research so I really played on that and how I had an international internship, senior project, eagle scout, etc. To anyone applying again I would basically just say try and concentrate a lot on the broader impacts and try and reiterate almost word for word each of the things they list in the instructions as to what they look for in the broader impacts. I did just that and made sure I covered everything.

[Jimbo2]

i wish they would give you your percentiles and z scores. not really sure what the harm would be -- maybe you could tell if someone reviewed your application and another person's based on an identifying z score number? Tough luck for the people that got a bunch of Es and no award. Boise State syndrome -- do perfect and it won't matter. you were screwed from the beginning because the strength of your schedule (or harshness of reviewers) wasn't tough enough, and it was always out of your control

Edited April 2, 2012 by Jimbo2

[Pitangus]

Looking at other people's reviews, this whole process seems even more arbitrary. I got E/E VG/VG VG/E which resulted in an HM. None of the comments said anything even remotely negative. I see other people in my discipline with equivlaent or even worse responses. I guess things like high school location, gender, and difficulty of reviewer really matters a lot.

I found this Reviewer's Guide to be the most informative in regards to the scoring process (at least as it was in 2008):

http://www.soest.haw...fo_NSF_GRFP.pdf

**thanks to vertices for posting it earlier in the thread**

Starting on page 11, it shows the numerical rating scale behind the E,VG,G,F,P ratings that we applicants get to see.

It also explains the standardized scoring (z-scores) that is supposed to combat the effects of varying reviewer difficulty (pg 13).

According to this guide, applicants were divided into 4 "quality groups:"

Applicants in Group 1 are all awarded fellowships

Applicants in Group 2 "receive awards to the limit of funds available using criteria such as geographical region, discipline, and other factors" (pg 16); the rest receive HMs

Applicants in Group 3 get HMs

Applicants in Group 4 do not get awards or HMs (this group includes applications that were below the 65th percentile after two ratings and were retired before the third rating).

From what I read it seems, in 2008, applicants were ranked and placed into the quality groups according to their z-score averages, but the panels of reviewers were able to deliberate and change ranks without changing scores (pg 14). So there is definitely some mystery surrounding the deliberation aspect of the rankings.

Plus, once the rankings are submitted, who knows what "other factors" the NSF uses to give additional awards to applicants in Group 2.

I think the combination of z-scores (which applicants don't see) and the reviewing panel's ability to change the rankings after scoring helps explain why the feedback we get (just the letter ratings and some comments) is often not helpful in determining why we did/didn't get an award/HM.

Edit: Jimbo's comment above coincides with this as well; I agree it would be nice to get at least the z-scores.

Edited April 3, 2012 by Pitangus

[Homo studentsis]

Z-scores would be nice.

My reviews were:

VG/VG

E/VG

VG/VG

with very little in terms of criticism in my reviews, and didn't even get an HM.

[rsbarner]

I found this Reviewer's Guide to be the most informative in regards to the scoring process (at least as it was in 2008): http://www.soest.haw...fo_NSF_GRFP.pdf **thanks to vertices for posting it earlier in the thread** Starting on page 11, it shows the numerical rating scale behind the E,VG,G,F,P ratings that we applicants get to see. It also explains the standardized scoring (z-scores) that is supposed to combat the effects of varying reviewer difficulty. According to this guide, applicants are divided into 4 "quality groups:" Applicants in Group 1 are all awarded fellowships Applicants in Group 2 "receive awards to the limit of funds available using criteria such as geographical region, discipline, and other factors" (pg 16); the rest receive HMs Applicants in Group 3 get HMs Applicants in Group 4 do not get awards or HMs (this group includes applications that were below the 65th percentile after two ratings and were retired before the third rating). From what I read it seems, in 2008, applicants were ranked and placed into the quality groups according to their z-score averages, but the panels of reviewers were able to deliberate and change ranks without changing scores (pg 14). So there is definitely some mystery surrounding the deliberation aspect of the rankings. Plus, once the rankings are submitted, who knows what "other factors" the NSF uses to give additional awards to applicants in Group 2. I think the combination of z-scores (which applicants don't see) and the reviewing panel's ability to change the rankings after scoring helps explain why the feedback we get (just the letter ratings and some comments) is often not helpful in determining why we did/didn't get an award/HM. Edit: Jimbo's comment above coincides with this as well; I agree it would be nice to get at least the z-scores.

Thanks for the info. That makes sense...I was wondering how my ratings looked so promising but I didn't even make it to a third reviewer.

[vertices]

I agree that it seems very arbitrary. I got a G/VG, G/G, and VG/VG and I got one. (Computer Science - other, bioinformatics) I'm guessing that the other category of bioinformatics helped and my broader impacts is really good although my academics aren't the strongest (mediocre GPA). I had heard from past people who got it that the broader impacts was almost more important than the research so I really played on that and how I had an international internship, senior project, eagle scout, etc. To anyone applying again I would basically just say try and concentrate a lot on the broader impacts and try and reiterate almost word for word each of the things they list in the instructions as to what they look for in the broader impacts. I did just that and made sure I covered everything.

I agree with bgodbgg on broader impacts.

I think it may be that we as students spend most of our time on our intellectual merit, even if we do spend a lot of time on broader impacts. That balance comes out in the applications. However, the weighting between the two of them is the same for the application.

Also, perhaps to the reviewers, intellectual merit looks almost the same amongst most applications, so it's the broader impacts where students get to differentiate themselves. I wonder if this also comes up in the LoRs. I sent my LoR writers the criteria as well as information that included my broader impact activities at their institution. I don't know if they spent any time discussing the broader impacts side but I wouldn't be surprised if lots of LoRs can speak to intellectual merit but might not have as much to say about broader impacts. For the future, discussing these matters with LoR writers may help an application.

[vertices]

I found this Reviewer's Guide to be the most informative in regards to the scoring process (at least as it was in 2008):

http://www.soest.haw...fo_NSF_GRFP.pdf

**thanks to vertices for posting it earlier in the thread**

Starting on page 11, it shows the numerical rating scale behind the E,VG,G,F,P ratings that we applicants get to see.

It also explains the standardized scoring (z-scores) that is supposed to combat the effects of varying reviewer difficulty.

According to this guide, applicants were divided into 4 "quality groups:"

Applicants in Group 1 are all awarded fellowships

Applicants in Group 2 "receive awards to the limit of funds available using criteria such as geographical region, discipline, and other factors" (pg 16); the rest receive HMs

Applicants in Group 3 get HMs

Applicants in Group 4 do not get awards or HMs (this group includes applications that were below the 65th percentile after two ratings and were retired before the third rating).

From what I read it seems, in 2008, applicants were ranked and placed into the quality groups according to their z-score averages, but the panels of reviewers were able to deliberate and change ranks without changing scores (pg 14). So there is definitely some mystery surrounding the deliberation aspect of the rankings.

Plus, once the rankings are submitted, who knows what "other factors" the NSF uses to give additional awards to applicants in Group 2.

I think the combination of z-scores (which applicants don't see) and the reviewing panel's ability to change the rankings after scoring helps explain why the feedback we get (just the letter ratings and some comments) is often not helpful in determining why we did/didn't get an award/HM.

Edit: Jimbo's comment above coincides with this as well; I agree it would be nice to get at least the z-scores.

This is a great overview Pitangus.

There's a lot of information we're missing because E and VG are so very broad. An E can be 40 or 50. A VG can mean 30 or 39. Therefore, someone with an E worth 40 raw points may change places with a VG of 39 raw points by quite a margin when z-scores are applied.

As for the re-ranking without changing scores, I've heard stories that reviewers will champion applications they feel strongly about at this stage, so I think that sometimes making a strong impression on one reviewer can overcome mediocre scores from other reviewers.

[Pitangus]

@vertices

I think that both of your points are possible, based on the information in that guide.

Even if things have changed since 2008, there is definitely more to the scoring and ranking procedure than the E,VG,G,F,P scores suggest.

I am curious as to why we get our results on that scale, since it seems that people with pretty much any mix of Es/VGs/Gs can end up with any of the award results. Not very informative.

If the reviewers used a 1-50 scale during the actual scoring, why can't we see those numbers?

Not that having the number scale would clear up the confusion, because there is still the question of z-scores and whether reviewers advocated for certain applicants and changed the rankings within their panel (the guide suggests that each panel was free to hold deliberations however they wanted during the ranking process).

It's a very involved and panel-dependent system, however it works, and I really feel for the people who are trying to figure out how to make their great applications better somehow. I am very lucky that the process somehow worked out for me.

[Homo studentsis]

Thats crazy!

IM / BI:

E/ E

VG/E

No award not even HM.

The most ridiculous part of this is that you didn't even get a 3rd reviewer. That means that you weren't even in the top 65% of applicants in your field. Given that you have 3 "excellents" and a "very good," it looks to me like you got screwed by having reviewers who tended to score applicants higher. You pretty much had no chance to get to a third reviewer, short of writing absolutely perfect essays.

[AnnzPB]

IM / BI:

1) E / VG

2) E / E

3) E / VG

Offered award

I will start grad school this fall. After reaching out to several previous winners for advice and feedback, I really made sure to emphasize BI in my application. Thankfully, my research experiences and academic record are very strong--as they are for many applicants--and so I knew that BI would be linchpin. My comments were pretty much all praiseworthy; one reviewer commented on how my application would foster diversity within the field (I'm an older South Asian female who made a career change from business into the social sciences) and that I compellingly demonstrated a commitment to outreach through several specific activities.

Regarding the point that some folks have already made regarding the importance of letters of recommendation explicitly addressing BI, I couldn't agree more. I coached all my letter writers on specifics to hammer home for BI regarding my research proposal and outreach activities. One of the reviewers picked up on this in a way that indicates that my weakness in addressing BI (as perceived by him) was compensated for by what my letters said--"The proposed research has the potential for significant societal impact. The application would be strengthened by a more elaborated discussion of some specific impacts, though the recommendation letters made such potential impacts clear. The application proposes to integrate two challenging and often misunderstood areas of research, so the applicant might consider how that unique strength could lead to benefits for society."

Edited April 3, 2012 by AnnzPB

[Dynamom]

E/VG

VG/VG

VG/VG

=HM

I would actually say those scores "look like" an HM. I'm too chicken to actually read what they wrote at this point - maybe after a glass of wine.

Although I'm feeling encouraged to try again, I'm not really sure how, between work, school, and raising 4 kids, I'm going to fit in any extra outreach.

[AnnzPB]

Has anyone on this thread ever dealt with making a change to their minor field of study once they were awarded? I have a question regarding this that I am very concerned about.

I applied as someone not yet in grad school and specified "psychology - other" as my minor field of study since my proposal and previous experiences didn't really fit neatly into any of the listed options for minor field of study. At the same time, I applied to clinical psychology Ph.D. programs for graduate school (my application was not clinically-oriented at all, hence why I was still able to be awarded). I also specified my undergraduate institution as my proposed graduate institution since that helped me make the strongest connection between my background and a potential research project, as well as the fact that my recommenders would be able to speak best to a project that already fit with what their specialties were.

I know several students (more than I can count) who have gotten the NSF even though they are enrolled in clinical psychology Ph.D. programs. They basically took the approach of making their application sound as "un-clinical" as possible, couching their previous experiences and proposed research in terms of basic science work. For example, several of them research ADD/ADHD, and so they wrote their applications under "cognitive psychology"; a few others I know who research marital and family clinical outcomes applied within "social psychology", etc etc.

In any case, now that I am making a decision on where to attend grad school, I am concerned about the fact that I will need to make a change to my minor field of study in order to accept the award since I am leaning towards a clinical psychology Ph.D. program that is 1) housed within a medical center, 2) very "clinically oriented" in terms of the types of research that it specializes in.

From what I gather, this is handled through the Coordinating Officer (CO) at the school that you'd wish to attend. Seems as if it is up to the CO's discretion as to whether they approve the change or not. Has anyone had to go through this before and run into an trouble? How much finagling/justifying do you think I would need to go through in order to iron everything out smoothly given my circumstances?

I know I will have more follow-up questions on this depending on what you all have to say, so I will just leave it at that for now. Y'alls input would be SOOOOOO appreciated. Thank you to you all. =)

Edited April 3, 2012 by AnnzPB

[vertices]

@Pitangus

While I'm curious about the raw scores, z-scores and rankings as well, I don't think they would be very helpful. The reason we're given the review sheets is so we can grow to produce strong applications (or other proposals, as this is an exercise to make us better at a wide range of such things in the future). The real problem seems to be that for many applicants the comments don't explain the rating in a way that the applicant can process and learn from.

I'm guessing that some reviewers see it as giving points rather than taking points a way. Therefore, they only write positive comments--the one's corresponding to the points they gave. They don't write down the points you didn't get from them and why, so the applicants don't know the ceiling they're trying to reach and how far they are from it. They don't know what the reviewer would write for the highest scoring applications.

To help with this point, I used the rubric developed by Robin Walker. The Missouri site with this information is down for a redesign, but the guide sheets are available from Illinois:

http://www.grad.illi...guidesheets.pdf

The rubric is on page 12 but the entire thing is super helpful.

Edited April 3, 2012 by vertices

[rsbarner]

The most ridiculous part of this is that you didn't even get a 3rd reviewer. That means that you weren't even in the top 65% of applicants in your field. Given that you have 3 "excellents" and a "very good," it looks to me like you got screwed by having reviewers who tended to score applicants higher. You pretty much had no chance to get to a third reviewer, short of writing absolutely perfect essays.

Yea...their lenient reviews do not help much as far as knowing what to fix when I reapply.

Edited April 3, 2012 by rsbarner

[justforkicks]

E/E

E/E

E/VG

awarded, second year applying.

I am in an Epidemiology PhD. Anyone else on this thread get an award in a public-health field? Just curious.

[colchiques]

No this is not necessarily true. I am a second year awardee and I only had a couple of conference papers "published" but nothing in a peer reviewed journal (I did have one submitted for peer review and I noted that in my application). A couple of the con's pointed out were my lack of publications but my IM and BI outside of that was so strong I still got great reviews. You just have to compensate for a lack of pubs in other ways

Same here. I have no pubs but a few conference presentations to try to make up for it. I thought this would totally tank me as a 2nd year, but I got an offer. My ratings had the same proportion of g/vg/e as my HM from last year, just shuffled. What my reviewers pointed out this time around: a strong/interesting proposal (improved using comments from last year), showing leadership with BI (essentially same as last year), and awesome letters of rec (same as last year). From this, I stand by my impression that in the end it's kind of a crapshoot and it can't hurt to try, try again.

[AnnzPB]

Sorry folks, just wanted to bump up my post again. Would REALLY appreciate it if y'all could give some input on my conundrum. Thank you so much.

Has anyone on this thread ever dealt with making a change to their minor field of study once they were awarded? I have a question regarding this that I am very concerned about.

I applied as someone not yet in grad school and specified "psychology - other" as my minor field of study since my proposal and previous experiences didn't really fit neatly into any of the listed options for minor field of study. At the same time, I applied to clinical psychology Ph.D. programs for graduate school (my application was not clinically-oriented at all, hence why I was still able to be awarded). I also specified my undergraduate institution as my proposed graduate institution since that helped me make the strongest connection between my background and a potential research project, as well as the fact that my recommenders would be able to speak best to a project that already fit with what their specialties were.

I know several students (more than I can count) who have gotten the NSF even though they are enrolled in clinical psychology Ph.D. programs. They basically took the approach of making their application sound as "un-clinical" as possible, couching their previous experiences and proposed research in terms of basic science work. For example, several of them research ADD/ADHD, and so they wrote their applications under "cognitive psychology"; a few others I know who research marital and family clinical outcomes applied within "social psychology", etc etc.

In any case, now that I am making a decision on where to attend grad school, I am concerned about the fact that I will need to make a change to my minor field of study in order to accept the award since I am leaning towards a clinical psychology Ph.D. program that is 1) housed within a medical center, 2) very "clinically oriented" in terms of the types of research that it specializes in.

From what I gather, this is handled through the Coordinating Officer (CO) at the school that you'd wish to attend. Seems as if it is up to the CO's discretion as to whether they approve the change or not. Has anyone had to go through this before and run into an trouble? How much finagling/justifying do you think I would need to go through in order to iron everything out smoothly given my circumstances?

I know I will have more follow-up questions on this depending on what you all have to say, so I will just leave it at that for now. Y'alls input would be SOOOOOO appreciated. Thank you to you all. =)

[Pitangus]

While I'm curious about the raw scores, z-scores and rankings as well, I don't think they would be very helpful. The reason we're given the review sheets is so we can grow to produce strong applications (or other proposals, as this is an exercise to make us better at a wide range of such things in the future). The real problem seems to be that for many applicants the comments don't explain the rating in a way that the applicant can process and learn from.

I agree that the rating sheets are meant to be helpful for future applications/proposals, and they can be in cases where the comments provide constructive criticisms or suggestions.

I'm just not seeing the usefulness of including the E/VG etc scores. If the point is to give applicants an idea of how well they did in terms of the scoring procedure, why not use the numerical scale if that's what reviewers really use?

Yes, those scores still aren't that useful for improving an application, but maybe they'd cause less confusion/frustration.

I like your idea of reviewers adding points rather than detracting based on some idea of "best."

It makes sense considering the way most reviewers seem to be commenting (mostly positives), though it requires applicants to think backwards, looking not just for aspects that were criticized but also ones that weren't praised (but perhaps did receive praise in other people's applications).

[Matba]

I wouldn't feel discouraged about applying for next year. I didn't even make it to the third round of reviews my first two times, and got the award this year. If you got HM you're very close. It's true you can't redo your undergrad, but if that really sunk you, you wouldn't even have HM. If you're getting good grades in your graduate program, that will help.

If there is a reason you struggled in undergrad, you can make that a focus of your personal statement and actually spin it into a positive. For example if you had an (undiagnosed?) learning disability, you can talk about how that makes you more aware and sensitive to the struggles of students with disabilities. Make sure you take actual, positive action based on this - follow the rule of show, don't tell. In this example, you could work with your services for students with disabilities office to identify accessibility barriers, and make changes that allow a student with a disability to conduct undergraduate research.

This is the big change I made between my prior, unsuccessful applications. Instead of trying to downplay weaknesses in my undergrad, I cast them in a positive light and took action based on this. I don’t have my review sheets back, so I can’t say whether this is what made the difference or not, though.

That's an awesome idea! Thank you and congrats on being offered an award!

Ps. Have you not seen the link on how to get to the reviewer comments? Someone posted it here. Or you can just wait until they officially come up tomorrow.

[trytrytry]

Interestingly I knew someone that had 4 reviewers last year and got an award… tie breaker round??

I got

VG/F

VG/G

VG/G

And nothing!

No comments (good or bad) on my proposal from any of the reviewers.. I don’t know whether it’s good enough to use next year or not. Obviously need to strengthen my broader impacts, but not bad for a first try.

[GNC]

I got

IM/B

E/E

G/G

VG/E

Got nothing. I am honestly really annoyed with the second reviewer and have no idea what the heck he/she was thinking. The reviewer raved about my intellectual merit with absolutely no complaint and gave me a G? Basically gave me a score of G/G without any explanation for IM and only one tid bit for BI.. what the heck am I supposed to do with that?

I guess it is hit or miss with the reviewers..

Edited April 3, 2012 by GNC

[irugga]

That's an awesome idea! Thank you and congrats on being offered an award!

Ps. Have you not seen the link on how to get to the reviewer comments? Someone posted it here. Or you can just wait until they officially come up tomorrow.

I didn't see the link, but I did get the official reviews. E/E E/E E/E . I can confirm that the reviewers approve of the 'learning from struggles' and taking action to help others based on this. Every reviewer spent most of the broader impacts reivew talking about this. Overall my approach was "This was a problem, but isn't anymore. Dealing with these health issues made me more aware/sensitive to students who have various struggles. Here is how I have worked with XY and Z to increase accessibility .Here is what I currently have in the works, and I plan to do this other thing."

One other thing I should mention is that I asked my undergraduate research advisor to include in his statement how I did have these health difficulties, but how we were able to work around them. I also asked my current advisor to include something about how it isn't interfering with my graduate work. I don't know if these things were important to do or not.

[vertices]

IM/BI

E/E

E/E

E/VG

Offered Award

The one VG said I could strengthen my explanation of how to integrate education/diversity in my proposal. I did have a paragraph with specifics on education, but I took out a bit tying it back to my diversity activities since it didn't flow right. I guess that was a bad idea. Another issue could be that this plan was near the end of the proposal and after the 'aid science' portion in my BI heading. Maybe the reviewer was barely reading by then, thinking that half of the BI heading would be more of the same. Also, my more general advance science/help scientists BI was mentioned in the first few sentences of the opening (which the reviewer pretty much quoted in my review), I did not put a mention of the educational extensions in the first paragraph. I think doing this would help combat reviewer fatigue and make it seem like the facet was explained more/throughout the essay.

What's interesting is that one reviewer spoke almost only about my proposal and recommendation letters. Another mentioned neither and spoke about my record, research experience and personal statement!

One reviewer said my proposal was well-written, indicating an ability to communicate research findings. I finished the first draft of my proposal with about three weeks to spare and did spend quite a bit of time rewriting the meat of it. I also used bold section starters for each section they asked for. Lastly, I noticed that the panel groups in computer science are quite broad when it comes to subfield, so I backed off of some of the technical details and made sure I explained/defined terms and ideas. I know in other areas (e.g. Life Sciences) the panels are more focused, so it may require a different strategy there.

I was fortunate to have a reviewer suggest improvements and to have E/Es that didn't get z-score filtered (too far) down.

Edited April 3, 2012 by vertices

[firefly luciferase]

IM/BI, offered award:

E/VG

E/E

E/E

The first reviewer couldn't seem to decide whether I was a boy or girl. The second reviewer's comments were more of a summary. The third reviewer, based on his/her summary of the major BI points from my personal statement, apparently had sub-par reading comprehension.

For those preparing for next year, the only helpful critiques were that my proposal wasn't clear on anticipated research results, and that it didn't address "resource access and constraints" - although I did specifically name professors at my school with expertise in each of the various aspects of my proposal. Apparently my PI helped by addressing more of these in the reference letter, but I thought this was a weird comment. One reviewer encouraged me to go to more conferences (I'm only a first year, so I haven't yet). 2 reviewers mentioned the strength of my reference letters for IM, so choose your writers carefully!

I went through about 3-6 iterations of each essay, and each one was read by several past GRFP winners, along with my advisor and friends who are in science but in unrelated fields. Another thing that I found really helpful was having copies of past winners' essays and score sheets, as I basically used this as a template to figure out what was important to emphasize.

Edited April 3, 2012 by firefly luciferase