NSF GRFP 2010-2011

[dendarii917]

Just remember folks, the most likely outcome is a no HM reject. It'll hurt less if you don't get excited about it.

Yes, statistically the most likely outcome is a no-HM rejection. Will letting the fantasy of being a fellow play out in my head make it more painful if I get a rejection email, as is extremely likely to happen? Of course it will... for a little bit.

That's a price I'm willing to pay for having been excited about it, though. And when/if that happens, I'll be sad and disappointed for a little bit, and then I'll remind myself of the following:

1.) My success in grad school will be determined by how hard I work, the quality of the research I do, and the quality of the people I get the privilege to learn from and publish with. The GRFP will ultimately not have too much impact on that.

2.) I learned a heck of a lot about proposal-writing from the application process. My application next year will be better. More importantly, since I plan to stay in academia, future NSF grant proposals of mine will be better because of what I learned this year from the GRFP.

3.) I will learn something from the rejection, too. Life in academia means having a thick skin: You will have conference papers you thought were groundbreaking rejected because one reviewer couldn't be bothered to read carefully and the review will state "doesn't address X" when in fact there's a whole section about X. You will get crappy evaluations from students who hated you no matter how hard you tried to help them. You will have your exciting, novel grant proposals rejected by the NSF, etc. over and over again, and meanwhile, see the guy who runs the lab down the hall who has never published anything that has advanced the state of the field, ever, bring in millions of dollars of funding because... lots of reasons. It's just the way it goes. I want to make it in academia because I love teaching, and the fact is that getting used to the rejection will be the hardest part for me. I value the opportunity to begin learning to shrug and move on.

4.) The process is hugely arbitrary. I spent much of yesterday paging through all the previous years' threads trying to find the exact announcement times, and I read pages of people posting their review scores and decisions. People with E/E, E/E, and E/VG get honorable mention. Someone in the same major field with VG/G, VG/VG, VG/VG will get a fellowship. Why? Maybe because the second person had harder reviewers, and the computer adjusts for whether a reviewer tends to grade harshly or not. Maybe the NSF needed more diversity in that field that year (be it gender, state representation, etc.), so pulled someone up who fit the criteria they need to satisfy their mission from Congress. There's always the chance you get screwed by some reviewer because he happens to hate your research area and thinks it's trendy, overfunded, and ultimately not going anywhere, because he just happens to research a competing solution. Or because he post-doc'ed with your advisor way back when and they hated one another. Academia is the most political place in the world. The point is, we'll never know.

I'm not saying it's not a huge achievement to win, because it definitely is! But if I win, I'll remember how arbitrary it is, and how many equally- to better-qualified candidates got rejections. I'll be hugely proud of myself, but I'll also know that the same application in a different year, or with different reviews, might have had a different outcome. And I'll be humble and know I got lucky, too. And if I get rejected, I'll be reminding myself the same thing.

So yes, the most likely outcome is a no-HM rejection. And yes, it would hurt less if I didn't get excited about it. But I'm still going to savor the excitement of that glimmer of a possibility anyway, because it IS exciting, and because I know that I'll walk away _better_ from this process either way the decision goes.

*shrug*

[gosset]

so the size of the Excel sheet on Fastlane just increased from 101 bytes to 123 bytes. There are no new rows, and I don't notice any other changes... but somebody must be working on something...

[violachica]

People with E/E, E/E, and E/VG get honorable mention. Someone in the same major field with VG/G, VG/VG, VG/VG will get a fellowship.

Yup I was one of those E/E E/E E/VG HMs while someone I know was a VG/VG VG/VG VG/G and got a fellowship. it does happen

That said, if i could "like" your post, I totally would. I think most of us know that realistically our chances are somewhat slim, but there's nothing wrong with having a little hope.

[violachica]

so the size of the Excel sheet on Fastlane just increased from 101 bytes to 123 bytes. There are no new rows, and I don't notice any other changes... but somebody must be working on something...

nope still 101 bytes when I opened it just now. maybe it depends on the browser?

[gosset]

nope still 101 bytes when I opened it just now. maybe it depends on the browser?

Maybe it depends on location? I'm in the same browser, and it was definitely (according to my download manager) 101 bytes all morning. I went back to my office to attempt and it's definitely 123 on the same computer (laptop), same browser. How odd.

[violachica]

Maybe it depends on location? I'm in the same browser, and it was definitely (according to my download manager) 101 bytes all morning. I went back to my office to attempt and it's definitely 123 on the same computer (laptop), same browser. How odd.

interesting, now in the same browser i used before it's 123

[uksawfly]

I think most of us know that realistically our chances are somewhat slim, but there's nothing wrong with having a little hope.

I couldn't agree more. Without hope, in a way, wouldn't all of what we in science do be pointless? We hope to make the world a better place with what we find, even if, in the end, only three other people in the world care a whit about our work.

EDIT: Also, to those of you working on the Excel file...more evidence for a Monday announcement?

Edited April 3, 2011 by uksawfly

[gosset]

EDIT: Also, to those of you working on the Excel file...more evidence for a Monday announcement?

I think it is pretty suggestive, but the real answer will come ~11 tonight when we see if they go down for maintenance (which is what happened last year). I think after the 2007(?) debacle, they've decided to just shut everything down while the data gets posted and re-launch when they're ready.

[aventinus]

I think it is pretty suggestive, but the real answer will come ~11 tonight when we see if they go down for maintenance (which is what happened last year). I think after the 2007(?) debacle, they've decided to just shut everything down while the data gets posted and re-launch when they're ready.

I'm not so sure. The 123 byte version has two extra columns compared to the 101 byte version (award year, award type). I don't know if this means anything.,,

[ChemistryGirl]

what excel file. how do i look???

[anotherflunky]

Yup I was one of those E/E E/E E/VG HMs while someone I know was a VG/VG VG/VG VG/G and got a fellowship. it does happen

In the aggregate, more E/E E/E E/VG will get a fellowship than not, but there's nothing surprising about an individual case getting an HM. The decision is based on a numerical scores that are not shown to you (go look at an empty rating sheet). It's just one number combining IM and BI and that's what gets normalized, not the letter ratings. The E/VG/G/F/P is just to indicate what the reviewer thought your weak point was i.e. if you got E/VG, then BI was weaker than IM. It's much less useful for comparing yourself with everyone else since the same reviewer might give different numerical scores to different E/E's.

[etale]

Just remember folks, the most likely outcome is a no HM reject. It'll hurt less if you don't get excited about it.

Also, it appears that many people who do actually receive an HM one year think that they will automatically get a fellowship the following year. In fact, I have heard of several people being accorded an HM two years in a row, or even more surprisingly an HM the first year and a flat rejection the second. (In response to a direct inquiry, the NSF told me that no information about any previous applications is used in the current fellowship cycle.)

[violachica]

Also, it appears that many people who do actually receive an HM one year think that they will automatically get a fellowship the following year. In fact, I have heard of several people being accorded an HM two years in a row, or even more surprisingly an HM the first year and a flat rejection the second. (In response to a direct inquiry, the NSF told me that no information about any previous applications is used in the current fellowship cycle.)

oy really? it would never EVER occur to me to just automatically assume anything about the following year's application cycle. new applicants, new reviewers...

[etale]

oy really? it would never EVER occur to me to just automatically assume anything about the following year's application cycle. new applicants, new reviewers...

Right, but not everyone knows how the NSF chooses their fellows.

[dendarii917]

It's much less useful for comparing yourself with everyone else since the same reviewer might give different numerical scores to different E/E's.

I suspect that even those numerical scores, though, are not the whole picture. I'm sure people with higher numerical scores get HMs while people with lower ones get fellowships. Not all the time, but I'm sure it happens. We're not behind the scenes, but there are lots and lots of reasons this could happen, the biggest being that NSF does have a mandate to ensure diversity across a variety of factors. (We could argue all day about whether that's ultimately a good thing or bad thing for science, and to be honest I could argue either side quite passionately, but the bottom line is that that's how Congress wants it).

My point was really just that we're never going to have a "scientific" answer or understanding for why we end up in the bucket we do, because there are so many human factors, and that the inherent arbitrariness of the process is something to remember as consolation if you're disappointed in the outcome (and as humbling if you're thrilled with the outcome).

In fact, I have heard of several people being accorded an HM two years in a row, or even more surprisingly an HM the first year and a flat rejection the second.

Partly this is because it gets harder every year: they rank the undergrads against other undergrads, the 1st-year-grads against other 1st years, etc., b/c for example, no publications is a lot more acceptable for a senior in college than a 2nd-year PhD student, and they realize this.

But also, I think this backs up the certain arbitrary nature of the process: it really depends a huge amount on your reviewer, who is a human and has all the associated biases, the diversity profile of the other applicants that year, etc etc. Which means that none of us should be putting too much personal stake in this! We should all be proud of ourselves just for slogging through the application, and sticking out grad school even though the pay is total crap and the hours are long, because of the contributions we want to make to the world.

And in the meantime, we can keep obsessing about the spreadsheet mystery... though I tend to think that it doesn't mean much and it'll still be Tuesday at 1am most likely.

[husky]

what excel file. how do i look???

On Fastlane, where they post the winners.

https://www.fastlane.nsf.gov/grfp/AwardeeList.do?method=loadAwardeeList

[etale]

Personally, the only reason I'd like to know soon is so that I can try to use it as a bargaining chip to have my application reconsidered at the schools by which I was already rejected. I realize that this hope is a long-shot, but I figure that three years of external funding might have some pull in having an admissions decision reversed (should the fellowship be awarded to me this year!).

Edited April 3, 2011 by etale

[Joesh]

If anything I am just happy I have 2 more years to try again if I don't get it this year. I should be able to improve my application in many ways!

[illini2015]

So who's banking on a Monday morning at 1 AM release of the results? Who's in?

[violachica]

So who's banking on a Monday morning at 1 AM release of the results? Who's in?

I'm thinking Monday night down for maintenance --> Tuesday morning 1am release?

[husky]

So who's banking on a Monday morning at 1 AM release of the results? Who's in?

I have a gut feeling that it's going to be Friday. But I'll go ahead and jump on the Monday bandwagon.

[dendarii917]

I'm thinking Monday night down for maintenance --> Tuesday morning 1am release?

This. By this time tomorrow, I hope to see a "server maintenance coming at 11pm" message up on Fastlane.

I think there's almost no chance it will be overnight tonight, though. Of course, if the gov't shuts down, this might be the last weekend in awhile NSF employees are getting paid. Maybe they'll all work overtime this weekend... but I doubt it.

[illini2015]

So last time they were debating the budget, NSF could not release the # of awards until it was passed? The politicians are still debating about the budget, and most likely they won't reach a compromise until the last minute on the deadline, Friday, April 8th. Does that mean we won't know for a couple more weeks, the earliest being late next week (Apr.11)? If they release the awards in waves like the other year, I'm going to die.

[dendarii917]

So last time they were debating the budget, NSF could not release the # of awards until it was passed? The politicians are still debating about the budget, and most likely they won't reach a compromise until the last minute on the deadline, Friday, April 8th. Does that mean we won't know for a couple more weeks, the earliest being late next week (Apr.11)? If they release the awards in waves like the other year, I'm going to die.

At least from the rhetoric on the Sunday talking heads shows this AM, it's not looking like a given that they'll reach a compromise at all, at least not without a temporary shutdown...

In 2009 there was still budget "uncertainty", at least according to the NSF at the time (I don't remember when the stimulus actually passed, went into effect, etc., so not sure what "uncertainty" meant exactly), when they went ahead and did the first wave of awards.

So that, plus the fact that when people called this week they were told "within a week", makes me pretty confident they don't have to wait until after whatever happens with the budget on April 8. That they can at the very least award a conservative number of fellowships (leaving room for a few tens of millions of dollars of cuts in whatever compromise is eventually reached), and tack on more later once things are settled.

Since that's what they did in 2009, it seems likely that's what's coming early this week. But there's some chance they'll just award them all: none of the proposals currently floated by either side have huge NSF cuts (though they do have small ones).

[hello! :)]

Awhile back, I came across these notes from someone who had served on one of the NSF GRF review panels from a couple years ago. I don't remember where I found them and I haven't been able to find them again on the web... Maybe for whatever reason this person had to take them down. In any case, I'll post them here as "notes from an anonymous NSF GRF review panelist," since I think that it provides some very helpful insights into the whole process.

Thank you, Anonymous Panelist!

* * *

Notes after serving on the review panel for the NSF Graduate Research Fellowship Program

## Executive Summary

+ Fellowship applications in field of Mechanical Engineering are evaluated by panel of ME faculty. Remember your audience when your write.

+ Two criteria—intellectual merit and broader impacts—have equal weight this year

+ Roughly 15 minutes to read an entire application—make your point clearly and quickly.

Roughly 10% of those _who apply_ for these fellowships will receive them. The applicants are all amazing individuals.

## The Process

All of the applications are evaluated by a panel of engineering faculty from a variety of schools, including research and teaching schools. Applicants for the same field (e.g. Mechanical Engineering) are evaluated by the same panel. This year the mechanical engineering panel we participated in had more than 20 members, and evaluated roughly 400 applications. The applications are sorted by level: level 1 is for those who are in their final undergraduate year, level 2 is for those who have just started their graduate programs, and there are also levels 3 and 4. While all those who are in level 1, level 2, etc are evaluated simultaneously (with criteria appropriate to the level), the final decisions on who to fund are not done by level.

NSF has two basic criteria for evaluating the applications: intellectual merit and broader impacts. _They are weighted equally._ After a “calibration exercise” which is designed to arrive at a kind of panel-wide understanding of what would constitute intellectual merit and broader impacts, each application is read by two panelists and scored (out of 50) in each category. One panelist reading a single application takes 15-20 minutes. Panelists can not read any applications for which they have a conflict of interest.

At the end of these first and second reads, applications get two Z-scores, where

Z

= [(Application's Score) − (Mean Application Score for that Panelist)] /

(Application Standard Deviation for that Panelist)

The Z-score is created to adjust for the fact that some panelists score applications much higher (on average) than others. The average of the Z-scores is used to rank the applications. Applications in the top 35% of the ranking get a third reading, as do any applications that have a wide discrepancy on their Z-scores. (The discrepancies are identified by computer and by the panelists.) The remaining 65% of the applications are retired, meaning they get no further consideration. After the third reading, applications that have widely varying Z-scores are returned to the 3 panelists for additional discussion and a resolution.

Finally a new ranking is created. The top 20 or so in this ranking are in Quality Group 1—definite funding. (Notice that this is only 5% of the applications.) The next 40 or so are in Quality Group 2—honorable mention and possible funding. (The top of this group may get funded, depending on resources. Also, this group is mined for recipients of special focus awards, programs for under-represented groups, etc.) The next 40 or so are in Quality Group 3—honorable mention. The rest are in Quality Group 4 and don’t get an award.

## Criteria for Evaluation

Here are criteria we used in evaluating the applications for level 1. Keep in mind that each panelist develops their own criteria based on the panel discussion, so that not every panelist is going to use the same standards. However, they will give you the general ideas behind the ratings. Also, they may seem very harsh, but this turns out to be essential since all of the applications are very strong.

### *Intellectual Merit*

>#### Excellent

>> 1. The research proposal clearly describes truly innovative or transformative research. (Transformative research transforms the way the field or society will think about the problem.)

>> 2. The student is academically well-prepared to conduct the research. Outstanding letters of recommendation, good GPA, solid GREs. The GPA does not need to be 4.0, but should be good. The GRE’s I saw were not as high as I anticipated.

>> 3. The student has a clear passion for their work which comes across in their writing and their actions to date.

>> 4. The student has prior research or industry experience that demonstrated the ability to define, initiate, and complete projects with substantial independence. Avoid describing senior design projects or class projects, as they were not generally persuasive.

>#### Very Good

>> (2), (3), and (4) still there. Research is solid (more than incremental) but not transformative or truly innovative. Or, (1), (2), and (3) but not (4).

>#### Good

>> (2) and (3), research is solid, but no (4).

>#### Fair

>> (2) and (3). Research proposal is weak and student has little experience.

>#### Poor

>> Student is not well-prepared, research plan is ordinary and sketchy, and the student has failed to convey any passion for their work.

### *Broader Impacts*

Be sure to address this topic, as Broader Impacts is half of the score and many applicants who were Excellent in Intellectual Merit did not address this area sufficiently.

Also, be sure to realize that almost everyone who applies for these grants wants to teach at the college level. Wanting to be a teacher at the college level is not evidence of broad impact.

_The identity of an individual does not constitute a broad impact._ This was explicitly discussed at the panel and explicitly ruled out (by NSF) as a broad impact. The fact that you are a female, Hispanic, Native American, African-American, etc does not, in itself, qualify as a broad impact. Also, personal struggle (health/economic/family) does not constitute a broad impact. Whoever you are, you need the types of broad impacts discussed under “Excellent” below. However, if you are part of an under-represented group or have overcome substantial difficulties in getting to your current position, do put this information in your personal statement if you want it to be considered. After the proposals are ranked, those who fall into these categories in Quality Group 2 will be picked up for additional funding opportunities.

>#### Excellent

>> 1. Demonstrated record of substantial service to the community, K-12 outreach, commitment to encouraging diversity, etc. Straight leadership a plus, but most highly ranked applicants have ongoing outreach/service activities.

>> 2. Clear explanation of the broader impacts of the research. How will it affect society, and why should the government fund your project over someone else’s? If the project’s success would have huge impacts on its engineering field, it would fall a bit here and a bit in Intellectual Merit. (Different panelists had different views on this.)

>#### Very Good

>> (1) or (2) is somewhat weaker. (1) still has demonstrated record (not just “I will do...”) but the record is weaker, or (2) is still there but the impact is less dramatic.

>#### Good

>> Both (1) and (2) are present, but weak.

>#### Fair

>> (1) or (2) is completely missing, but the one that is present is at an Excellent level.

>#### Poor

>> (1) or (2) is completely missing, the one component that is present is only at a Very Good level.

* * *

Edit: Had to fix some dumb formatting issues.

Edited April 3, 2011 by hello! :)