NSF GRFP 2014-2015

[VentiHalfCaff]

So, in reading the NSF website, it seems that the three Peer Reviewers, who are awarding the assessments, basically determine whether your application will make it into the pile of "finalists," if you will, and then those "finalists," so to speak, are considered for further review by the NSF Program Officer.

 

Then, "After scientific, technical and programmatic review and consideration of appropriate factors, the Program Officer makes an award/decline recommendation to the Division Director."

 

And then, "Final programmatic approval for a proposal is generally completed at the Division level. If the Program Officer makes an award recommendation and the Division Director concurs, the recommendation is submitted to the Division of Grants and Agreements for award processing."

 

So, I'm just really curious about the different people involved. I'm new to this whole process and have very little knowledge of it.

 

Who might be the Peer Reviewers? Volunteers who work in the field in the applicant's area of "expertise?" So, mechanical engineering proposals might be reviewed by 3 mechanical engineers who actually work in the field? Or maybe by mechanical engineering professors? Grad students? A little of each?

 

And who might be the NSF Program Officer? Is there more than one NSF Program Officer to cover the spectrum of areas of study? Or is there one NSF Program Officer who reviews ALL applications in all fields that make it past the 3 Peer Reviewers? (That would be a heck of a lot of work!)

 

And finally, I have the same questions about the Division Director as I do about the Program Officer. 

 

I'm just really curious. 

 

I was awarded with a VG/VG, E/VG, VG/VG with lots of positive comments. My Peer Reviewers seemed to kind of like my proposal/application, but I guess they weren't totally wowed, or I would have received more E's. 

 

But, there are clearly some proposals (listed on this forum) that wowed their peer reviewers enough to earn a lot of E's, and many of those received HM's or were denied.

 

From reading more about the review process on the NSF website, it seems that the NSF Program Officer who reviewed my proposal may have been more impressed with it than my 3 Peer Reviewers. Or maybe my Peer Review assessments (lots of VG's) were some of the best in my group of senior undergrads in my field? 

 

Can somebody with some experience please comment? Thanks!

[Aliake]

So, in reading the NSF website, it seems that the three Peer Reviewers, who are awarding the assessments, basically determine whether your application will make it into the pile of "finalists," if you will, and then those "finalists," so to speak, are considered for further review by the NSF Program Officer.

 

Then, "After scientific, technical and programmatic review and consideration of appropriate factors, the Program Officer makes an award/decline recommendation to the Division Director."

 

And then, "Final programmatic approval for a proposal is generally completed at the Division level. If the Program Officer makes an award recommendation and the Division Director concurs, the recommendation is submitted to the Division of Grants and Agreements for award processing."

 

So, I'm just really curious about the different people involved. I'm new to this whole process and have very little knowledge of it.

 

Who might be the Peer Reviewers? Volunteers who work in the field in the applicant's area of "expertise?" So, mechanical engineering proposals might be reviewed by 3 mechanical engineers who actually work in the field? Or maybe by mechanical engineering professors? Grad students? A little of each?

 

And who might be the NSF Program Officer? Is there more than one NSF Program Officer to cover the spectrum of areas of study? Or is there one NSF Program Officer who reviews ALL applications in all fields that make it past the 3 Peer Reviewers? (That would be a heck of a lot of work!)

 

And finally, I have the same questions about the Division Director as I do about the Program Officer. 

 

I'm just really curious. 

 

I was awarded with a VG/VG, E/VG, VG/VG with lots of positive comments. My Peer Reviewers seemed to kind of like my proposal/application, but I guess they weren't totally wowed, or I would have received more E's. 

 

But, there are clearly some proposals (listed on this forum) that wowed their peer reviewers enough to earn a lot of E's, and many of those received HM's or were denied.

 

From reading more about the review process on the NSF website, it seems that the NSF Program Officer who reviewed my proposal may have been more impressed with it than my 3 Peer Reviewers. Or maybe my Peer Review assessments (lots of VG's) were some of the best in my group of senior undergrads in my field? 

 

Can somebody with some experience please comment? Thanks!

Unfortunately I cannot answer this, but rather I am wondering the same thing. I do believe that your status (graduating senior, 1st year, or 2nd year) does influence the weight placed on your proposal. I'm a undergrad senior and was awarded with E/VG, E/P, E/G... yes, the middle reviewer gave me a poor because I did not explicitly define and title a section "Broader impacts". 

 

But in their summary statements, the reviewers were overwhelmingly positive because of the fact that I had two first author pubs and was involved in a significant amount of STEM outreach. Ultimately, I think the proposal serves as a baseline (i.e. student scored X points on proposal so we'll look at the rest of the app), and that baseline threshold increases as you advance each year academically. Like the GRE, the proposal likely only sways minds in the most extreme cases such as 6 E ratings. Just my thoughts.

[VentiHalfCaff]

^ That's interesting, Aliake. I did read on the NSF site that most awardees do NOT receive all E's. From my reading, it seems that the Program Officer has much more to do with the selection than the reviewers, assuming that one meets the basic criteria to move past the Peer Review.

 

Anybody else have any insight or comments on the process?

[rufusjbuttlicker]

So should I be sending a "thank you again, I got an honorable mention" note to my letter writers? 

Yes.

you should let all of your letter writers know the outcomes of all of your fellowship/grad school applications.

You can include the reviews too if you like. As a curiosity, I know if I was a letter writer I would like to see it.

 

I'm planning on writing one big email at the end of the fellowship cycle informing them of the status of all I've applied for (8 of them).

[braindump]

So, in reading the NSF website, it seems that the three Peer Reviewers, who are awarding the assessments, basically determine whether your application will make it into the pile of "finalists," if you will, and then those "finalists," so to speak, are considered for further review by the NSF Program Officer.

 

Then, "After scientific, technical and programmatic review and consideration of appropriate factors, the Program Officer makes an award/decline recommendation to the Division Director."

 

And then, "Final programmatic approval for a proposal is generally completed at the Division level. If the Program Officer makes an award recommendation and the Division Director concurs, the recommendation is submitted to the Division of Grants and Agreements for award processing."

 

So, I'm just really curious about the different people involved. I'm new to this whole process and have very little knowledge of it.

 

Who might be the Peer Reviewers? Volunteers who work in the field in the applicant's area of "expertise?" So, mechanical engineering proposals might be reviewed by 3 mechanical engineers who actually work in the field? Or maybe by mechanical engineering professors? Grad students? A little of each?

 

And who might be the NSF Program Officer? Is there more than one NSF Program Officer to cover the spectrum of areas of study? Or is there one NSF Program Officer who reviews ALL applications in all fields that make it past the 3 Peer Reviewers? (That would be a heck of a lot of work!)

 

And finally, I have the same questions about the Division Director as I do about the Program Officer. 

 

I'm just really curious. 

 

I was awarded with a VG/VG, E/VG, VG/VG with lots of positive comments. My Peer Reviewers seemed to kind of like my proposal/application, but I guess they weren't totally wowed, or I would have received more E's. 

 

But, there are clearly some proposals (listed on this forum) that wowed their peer reviewers enough to earn a lot of E's, and many of those received HM's or were denied.

 

From reading more about the review process on the NSF website, it seems that the NSF Program Officer who reviewed my proposal may have been more impressed with it than my 3 Peer Reviewers. Or maybe my Peer Review assessments (lots of VG's) were some of the best in my group of senior undergrads in my field? 

 

Can somebody with some experience please comment? Thanks!

 

This makes it sound like the opinion of one person can completely override the opinion of three others? Somehow that seems a little wonky. I guess that's what got me, though. All E's & 1 VG, two "strongly recommend," and "one of the best proposals on this level," and I guess that one guy down the line didn't like something about it. That explains a lot, though, why some people with lower ratings can get it and vice versa. I thought it was all just based on the ratings alone. Sorry I didn't answer your question; just voicing my frustration. 

[VentiHalfCaff]

^ Hmmm. I don't agree that it sounds like one Program Officer's opinion can completely override three Peer Reviewers' opinions. The NSF website says, somewhere (I read it at one point), that the Peer Reviewers are NOT tasked with recommending or denying an applicant, but rather solely with the task of assessing each proposal using the prescribed Merit Criteria. The Program Officer uses their assessments and comments as a tool to more quickly weigh the propsals' merits, I think. Since the Peer Reviewers are not tasked to say yea or nay, their opinions aren't really being overridden. They're just independently assessing the proposals based on their merits, not giving an opinion on "pass/fail/HM." It's interesting that your comments said things like "strongly recommend" and "one of the best on this level," though. Mine didn't have those sorts of comments. The comments on mine were addressing my proposal as an independent unit, it seems, without any reference to any comparison value or any eye towards "passing or failing."

But your response got me reading more.

Here's one thing I found on the website:

"4. FACT: Most proposals that are awarded do not receive all "Excellents."

DISCUSSION: It is not true that a proposal must receive all "Excellents" to be funded; in fact, most proposals that are awarded do not receive all "Excellents." Furthermore, even if you get all "Excellents," you may not be funded. See the annual reports to the National Science Board on the National Science Foundation's Merit Review Process for data about proposals and success rates, as well as further information and data concerning the merit review process."

So the decisions are definitely not based on the assessments alone.

I also just learned that the reviewers are volunteers who have some expertise in the field and are chosen by the Program Officer, and that, yes, there are many Program Officers, one for each category (Life Sciences, Engineering, Psychology, etc).

This info is all from reading the website. It's always possible that I misinterpreted some stuff, but that's how the site read to me.

I would still love to hear more from anybody!

Edited April 1, 2015 by VentiHalfCaff

[geographyrocks]

From what I understand from talking with two different reviewers:

The applications are checked over for any obvious disqualifications.

The peer reviews take place after a big meeting amongst each division.  It's now done through video chat.  The reviewers decide if the person is worth funding.  Those that are deemed worthy go to the program officer.  The program officer then decides how many per category will be funded.  While most will tell you that the research isn't as important as the person, I have a feeling that the project proposed is important for decided who will/will not be funded.  Ex: they don't want to fund four people all doing the same type of project within a field.  I think they also try to get a mix between theory and application. 

I *think* the program officer also looks into the diversity of the applicants, but that may be the next step. 

After the list is narrowed again, diversity is taken into account. 

I think this is where a lot of E's are taken out.  I've also heard that there is some bias to a proposal being too well written because the reviewers suspect the majority of the ideas or writing came from an advisor. 

[braindump]

^ Hmmm. I don't agree that it sounds like one Program Officer's opinion can completely override three Peer Reviewers' opinions. The NSF website says, somewhere (I read it at one point), that the Peer Reviewers are NOT tasked with recommending or denying an applicant, but rather solely with the task of assessing each proposal using the prescribed Merit Criteria. The Program Officer uses their assessments and comments as a tool to more quickly weigh the propsals' merits, I think. Since the Peer Reviewers are not tasked to say yea or nay, their opinions aren't really being overridden. They're just independently assessing the proposals based on their merits, not giving an opinion on "pass/fail/HM." It's interesting that your comments said things like "strongly recommend" and "one of the best on this level," though. Mine didn't have those sorts of comments. The comments on mine were addressing my proposal as an independent unit, it seems, without any reference to any comparison value or any eye towards "passing or failing."

But your response got me reading more.

Here's one thing I found on the website:

"4. FACT: Most proposals that are awarded do not receive all "Excellents."

DISCUSSION: It is not true that a proposal must receive all "Excellents" to be funded; in fact, most proposals that are awarded do not receive all "Excellents." Furthermore, even if you get all "Excellents," you may not be funded. See the annual reports to the National Science Board on the National Science Foundation's Merit Review Process for data about proposals and success rates, as well as further information and data concerning the merit review process."

So the decisions are definitely not based on the assessments alone.

I also just learned that the reviewers are volunteers who have some expertise in the field and are chosen by the Program Officer, and that, yes, there are many Program Officers, one for each category (Life Sciences, Engineering, Psychology, etc).

This info is all from reading the website. It's always possible that I misinterpreted some stuff, but that's how the site read to me.

I would still love to hear more from anybody!

 

Hmm, this is very helpful, thank you! It does make sense that it isn't based solely on assessments; I just wish I knew what else they took into account. It'd make it easier to know what they're looking for and how I could refine my application. I really wonder what the process is like and how many applications a single person goes through. Honestly, I really expected not to win it; I felt my application was pretty weak at the time. But getting the great reviews and nothing out of it felt like getting winning lottery numbers only to find out they already gave the lottery away. 

[braindump]

I think this is where a lot of E's are taken out.  I've also heard that there is some bias to a proposal being too well written because the reviewers suspect the majority of the ideas or writing came from an advisor. 

 

That's really unfortunate for those who actually wrote their own and did a great job on their own. But I guess it's also unfortunate for those losing to someone who may not have written their own proposal. 

[VentiHalfCaff]

While most will tell you that the research isn't as important as the person, I have a feeling that the project proposed is important for decided who will/will not be funded.  Ex: they don't want to fund four people all doing the same type of project within a field.  I think they also try to get a mix between theory and application.

 

That's interesting. I hadn't heard before that "the research isn't as important as the person." If that's the case (and maybe it is), then that might partially explain why I was awarded with only 1 E and 5 VG's. If the focus is on the person or the person's potential rather than the details of the proposed research, then maybe that explains why a person might get "merely" above average assessments and still earn the award.

 

Don't get me wrong: I don't think my proposal was shabby in any way, and I'm not saying I wasn't qualified, (and I know many other people were qualified but weren't awarded), but the fact that I was awarded my with mostly VG assessments has me very curious about the process. If it's said by most that "the research isn't as important as the person," maybe the 3 Peer Reviewers are sort of responsible for analyzing the proposal itself -- mainly the two statements, with less emphasis on "the person" (the rest of the application) -- and thought that my proposal was Very Good, but maybe the Program Officer is responsible for more thoroughly viewing the application in its entirety and thought the "person" or the "potential of the person" was award-worthy? I know ... it's a long shot and a total guess. I'm thinking out loud. But could this be the case?

 

I *think* the program officer also looks into the diversity of the applicants, but that may be the next step. 

After the list is narrowed again, diversity is taken into account. 

I think this is where a lot of E's are taken out.  I've also heard that there is some bias to a proposal being too well written because the reviewers suspect the majority of the ideas or writing came from an advisor.

 

Wow. I have to disagree with you about a lot of the E's being taken out after considering diversity. That sounds to me like a very narrow view. It's as if you're implying that the non-diverse people naturally have the lion's share of E's, and that the diverse people naturally have lower non-E assessments ... but are awarded anyway. It's kind of like the old line, generally uttered by white people, of URM's, in the past, "He (or she) took my slot!" As if the "slot" belonged to the non-URM in the first place, or as if he was entitled to it from the beginning. Or as if the URM was not qualified by his or her own merits in the first place -- which non-URM's seemed to assume based solely on the URM's gender or the color of his/her skin. It is my bet that there are plenty of E's and VG's to go around -- from URM's and non-URM's, and that there were other possibly minor differences, besides diversity, that separated an awardee from an HM or deny. (And I'm speaking as a non-diverse, white male, non-URM, from a large state public who got only 1 E.)

 

Sure, I know that NSF is focused on Broadening Diversity and promoting a team of NSF-funded scientists and engineers in STEM fields that mirrors our population. I understand that diversity IS part of the equation. But I am guessing that it's more of an occasional tie-breaker than a factor that routinely tosses exceptionally well-qualified non-URM's aside in favor of less-than-qualified URM's. That's too narrow for my thinking.

 

Your other comment, geography rocks, is interesting. I hadn't heard that either. (I've really had very little exposure to the process.) Based on my knowledge of the college application process, and comparing this to that, I have an educated guess about this. The fact that a proposal is exceptionally well-written -- that fact alone -- may not be a threat to the applicant at all. Instead, the threat of having your proposal downgraded or denied for exceptional writing might arise only if the rest of your application doesn't support the fact that you're an exceptional writer. If your grades, test scores, academic honors, fellowships, scholarships, publications, presentations, and reference letters all support the fact that you can write exceptionally well, then there's no threat. But if your proposal is exceptionally well-written and the remainder of your application doesn't support the concept that you are an exceptional writer ... well, then, it kind of makes sense that there's a threat. (It did not occur to me that some people might turn in their advisor's work -- or that an advisor might "help" an applicant too much!) I do know that in the college application and scholarship process, admissions officers look for consistency across the entire application. If the entire application is consistent across the board in terms of stated strengths, the applicant is more likely to be selected for admission/scholarship. 

 

It does make sense that it isn't based solely on assessments; I just wish I knew what else they took into account. It'd make it easier to know what they're looking for and how I could refine my application. I really wonder what the process is like and how many applications a single person goes through. Honestly, I really expected not to win it; I felt my application was pretty weak at the time. But getting the great reviews and nothing out of it felt like getting winning lottery numbers only to find out they already gave the lottery away.

 

I understand, braindump. We have some of the same wonders. You had mostly E's and no award, and I had mostly VG's and an award. It's confusing. Of course, there's the fact that different Peer Reviewers are going to rate differently. I suppose one applicant could have easily-impressed Peer Reviewers while another has hard-ass Peer Reviewers. And of course, there are bound to be other variables. But I do understand your frustration. You got great comments and tippy top assessments, and yet you weren't awarded. Disappointing.

 

Anyway, it's been an interesting discussion. Lots of food for thought. Please keep those ideas coming. What have you heard about the process?

Edited April 1, 2015 by VentiHalfCaff

[gellert]

Someone on a previous page also said that an E is weighted more heavily if the reviewer in question mostly gives lower scores than an E from a reviewer who gives lots of VGs/Es out.  The source was fairly legit iirc, as well.

[VentiHalfCaff]

^Yeah, that makes sense. I see what you mean.

Edited April 2, 2015 by VentiHalfCaff

[fd.711]

So, in reading the NSF website, it seems that the three Peer Reviewers, who are awarding the assessments, basically determine whether your application will make it into the pile of "finalists," if you will, and then those "finalists," so to speak, are considered for further review by the NSF Program Officer.

 

Then, "After scientific, technical and programmatic review and consideration of appropriate factors, the Program Officer makes an award/decline recommendation to the Division Director."

 

And then, "Final programmatic approval for a proposal is generally completed at the Division level. If the Program Officer makes an award recommendation and the Division Director concurs, the recommendation is submitted to the Division of Grants and Agreements for award processing."

 

So, I'm just really curious about the different people involved. I'm new to this whole process and have very little knowledge of it.

 

Who might be the Peer Reviewers? Volunteers who work in the field in the applicant's area of "expertise?" So, mechanical engineering proposals might be reviewed by 3 mechanical engineers who actually work in the field? Or maybe by mechanical engineering professors? Grad students? A little of each?

 

And who might be the NSF Program Officer? Is there more than one NSF Program Officer to cover the spectrum of areas of study? Or is there one NSF Program Officer who reviews ALL applications in all fields that make it past the 3 Peer Reviewers? (That would be a heck of a lot of work!)

 

And finally, I have the same questions about the Division Director as I do about the Program Officer. 

 

I'm just really curious. 

 

I was awarded with a VG/VG, E/VG, VG/VG with lots of positive comments. My Peer Reviewers seemed to kind of like my proposal/application, but I guess they weren't totally wowed, or I would have received more E's. 

 

But, there are clearly some proposals (listed on this forum) that wowed their peer reviewers enough to earn a lot of E's, and many of those received HM's or were denied.

 

From reading more about the review process on the NSF website, it seems that the NSF Program Officer who reviewed my proposal may have been more impressed with it than my 3 Peer Reviewers. Or maybe my Peer Review assessments (lots of VG's) were some of the best in my group of senior undergrads in my field? 

 

Can somebody with some experience please comment? Thanks!

This is the review process for a full NSF grant, NOT the NSF-GRFP. (Among other clues, the GRFP isn't administered by the Division of Grants and Agreements, it's administered by the Division of Graduate Education. Also, NSF would never refer to GRFP reviewers as "peer" reviewers, because the reviewers--faculty and other experts in the various fields--are NOT the applicants'--current and future grad students--peers. They're our superiors!)

 

The info in this post therefore may or may not be applicable to the GRFP review process. From the program solicitation, this is the review process specifically for the GRFP:

 

Applications will be reviewed online by virtual panels of disciplinary and interdisciplinary scientists and engineers and other professional graduate education experts. Panels will review applications from broad areas of related disciplines. Applicants are reviewed in panels based on their selection of a primary Field of Study (see Fields of Study in Appendix). Selection of a primary Field of Study determines the application deadline and the panel that will review the application.

<snip>

Applications submitted in response to this program solicitation will be reviewed online by Panel Review.

The application evaluation involves the review, rating, and ranking of applications by disciplinary and interdisciplinary scientists and engineers, and other professional graduate education experts.

The primary responsibility of each panel is to evaluate the merit of eligible GRFP applications by applying the National Science Board-approved Merit Review Criteria of Intellectual Merit and Broader Impacts, and to subsequently recommend applicants for NSF Graduate Research Fellowships. Panelists are instructed to review the applications holistically in the context of applying NSF's Merit Review Criteria and the GRFP emphasis on demonstrated potential for significant achievements in science and engineering. NSF determines the successful applicants from these recommendations, with Fellowships and Honorable Mention offered based on the GRFP portfolio within the context of NSF's mission.

Edited April 2, 2015 by flyerdog11

[geographyrocks]

Wow. I have to disagree with you about a lot of the E's being taken out after considering diversity. That sounds to me like a very narrow view. It's as if you're implying that the non-diverse people naturally have the lion's share of E's, and that the diverse people naturally have lower non-E assessments ... but are awarded anyway.

That wasn't even remotely what I meant, but I could see how it could be taken that way. 

It's pretty time-consuming and difficult to write these proposals so I have to think that there are WAY more people with excellent reviews than there are awards to be given out (much like there are often amazing candidates that are rejected from grad school).  I am in no way saying that the ethnic lady with four fairs in her review is going to be bumped up to award winner over a white guy with all E's nor am I saying that minorities aren't capable of the same high scores.  What I am saying is that in each category you have a spread of very awesome candidates.  Since NSF is big on diversity, they will make sure that they aren't giving all 20 awards for that category to 20 white men.  So they might look at the applicant pool they have and choose a mix that represents the population.  Every single person in that pool is qualified.  They also make sure that there is a mix of winners from state schools and ivy schools. 

 

 

Based on my knowledge of the college application process, and comparing this to that, I have an educated guess about this. The fact that a proposal is exceptionally well-written -- that fact alone -- may not be a threat to the applicant at all. Instead, the threat of having your proposal downgraded or denied for exceptional writing might arise only if the rest of your application doesn't support the fact that you're an exceptional writer. If your grades, test scores, academic honors, fellowships, scholarships, publications, presentations, and reference letters all support the fact that you can write exceptionally well, then there's no threat. But if your proposal is exceptionally well-written and the remainder of your application doesn't support the concept that you are an exceptional writer ... well, then, it kind of makes sense that there's a threat.

That makes logical sense.  I doubt it happens that often, but who knows!

[isilya]

In response to reviewers care more about the person than the research, this is what I've always heard as well. There's a reason the NSF doesn't require you to follow through on your proposed project if you're funded: they care about your ability to conduct research and impact society more broadly, rather than the specifics of your research.

[fd.711]

In response to reviewers care more about the person than the research, this is what I've always heard as well. There's a reason the NSF doesn't require you to follow through on your proposed project if you're funded: they care about your ability to conduct research and impact society more broadly, rather than the specifics of your research.

Exactly. The GRFP is a fellowship, not a grant; that is, it funds YOU, not your research.

 

That said, your ability to develop (and concisely explain) a feasible, useful research program is a pretty important factor in determining whether or not you, as a researcher, should receive the funding. Even if you never conduct the project(s) you proposed in your application, you still need to demonstrate that you can come up with enough good ideas to keep you busy for the duration of the fellowship.

[VentiHalfCaff]

Wow. I love the discussion. Lots of good info.

 

This is the review process for a full NSF grant, NOT the NSF-GRFP. (Among other clues, the GRFP isn't administered by the Division of Grants and Agreements, it's administered by the Division of Graduate Education. Also, NSF would never refer to GRFP reviewers as "peer" reviewers, because the reviewers--faculty and other experts in the various fields--are NOT the applicants'--current and future grad students--peers. They're our superiors!)

 

Oh!! That makes sense! So glad you joined the discussion, flyerdog11. It sounds like you know stuff. Tell us more!

 

I mentioned in an earlier thread that I was merely reading stuff and trying to make sense of it -- and that I may have mis-interpreted what I was reading. Sounds like I did. But you sound like you may know a thing or two about the process. I'd love to hear more about it, if you have more to offer.

 

@geographyrocks, Okay. I see what you mean. I agree with that. It was your guess in the earlier post that "that's where a lot of E's are taken out" that got me writing.

 

@isilya, Interesting that you've always heard that, too. I like the way you put it: "There's a reason the NSF doesn't require you to follow through on your proposed project if you're funded: they care about your ability to conduct research and impact society more broadly, rather than the specifics of your research." And do you think, isilya, that the assessments (E, VG, G, etc.) are mainly assessing your Personal Statement and your Research Plan Statement? In other words, are they reflective of the 2 Statements, more than they are of your application as a whole? Any idea?

[fd.711]

I honestly can't say much about the review process beyond what's described in the program solicitation. (And even if I did know anything beyond NSF's official info, I'm not really supposed to know it.) Most of what I know comes from my own experiences as an applicant and the general advice I received about how to make a strong application.

 

I was awarded the GRFP last year (2nd of 2 attempts), and I'm happy to share my experiences here. But I'm afraid I don't have any more insight than you guys about the inner workings of the review process . I can only imagine that GRFP decisions are like laws and sausages (and journal articles): You're better off not knowing how they're made.

[Pitangus]

Someone on a previous page also said that an E is weighted more heavily if the reviewer in question mostly gives lower scores than an E from a reviewer who gives lots of VGs/Es out.  The source was fairly legit iirc, as well.

 

This gets brought up multiple times every year, and I'm probably starting to sound like a broken record to those who have actually read through the threads, but yes, this is how it has been done according to the actual reviewers' guide (in 2008 and I would guess it hasn't changed much):

 

A given subject panel distributed the applications so that each one went to two reviewers. The reviewers scored the application on a 1-50 scale for IM and BI (this scale corresponds to the P - E scores applicants see), and applications that scored below the 65th percentile after two reviewers were retired without being read by a third reviewer. The remaining proposals got a third reviewer, and were then ranked based on the average of the z-scores (standardized scores weighted based on all scores given by a reviewer, to help offset reviewer variability). The reviewers then deliberate to finalize the ranking.

 

Applications were then sorted into four Quality Groups by their ranks:

Applicants in Group 1 are all awarded fellowships

Applicants in Group 2 "receive awards to the limit of funds available using criteria such as geographical region, discipline, and other factors" (so this is probably where diversity comes in); the rest receive HMs

Applicants in Group 3 get HMs

Applicants in Group 4 do not get awards or HMs (this group includes applications that were below the 65th percentile after two ratings and were retired before the third rating).

 

So to conclude again: the scores applicants receive don't tell you much about how you were actually ranked, so it's pointless comparing how many Es and VGs you received compared to other applicants.

Edited April 2, 2015 by Pitangus

[braindump]

This gets brought up multiple times every year, and I'm probably starting to sound like a broken record to those who have actually read through the threads, but yes, this is how it has been done according to the actual reviewers' guide (in 2008 and I would guess it hasn't changed much):

 

A given subject panel distributed the applications so that each one went to two reviewers. The reviewers scored the application on a 1-50 scale for IM and BI (this scale corresponds to the P - E scores applicants see), and applications that scored below the 65th percentile after two reviewers were retired without being read by a third reviewer. The remaining proposals got a third reviewer, and were then ranked based on the average of the z-scores (standardized scores weighted based on all scores given by a reviewer, to help offset reviewer variability). The reviewers then deliberate to finalize the ranking.

 

Applications were then sorted into four Quality Groups by their ranks:

Applicants in Group 1 are all awarded fellowships

Applicants in Group 2 "receive awards to the limit of funds available using criteria such as geographical region, discipline, and other factors" (so this is probably where diversity comes in); the rest receive HMs

Applicants in Group 3 get HMs

Applicants in Group 4 do not get awards or HMs (this group includes applications that were below the 65th percentile after two ratings and were retired before the third rating).

 

So to conclude again: the scores applicants receive don't tell you much about how you were actually ranked, so it's pointless comparing how many Es and VGs you received compared to other applicants.

 

Very informative, thanks!

 

Oh well. At least they're good for an ego boost, hehe. 

[mheimbu2]

1st time poster here. I had been following this thread and wanted to ask a couple questions to fellow GRFP applicants. I should start by saying that that I was not recommended for a fellowship and this was my last and only attempt. I know there is no appealing the decision, and I am not really trying to, but I wanted to hear from some other people on the feedback they received and maybe get some advice.

While I was obviously disappointed that I did not receive an award I was pretty upset about one of the reviewers comments that strongly implied they had barely read my application. Here's some context.

My proposal was in the field of ecology and the project involved urban ecology specifically. Due to the nature and design of the project I would need to almost exclusively use private lands in my work. I proposed that I would gain access to private lands by speaking to local Audubon Societies (my project involved birds), conservation organizations, and neighborhood organizations, hoping to generate interest and get landowner's to volunteer me access to their property. These properties would be more like backyards in a suburban landscape. In my proposal I suggested that working in people's backyards would allow me to do outreach and education. This outreach and would be basically house-to-house and one-on-one by necessity. This is the feedback on my broader impacts, they gave me a "fair" assessment.

"The broader impacts included are traditional modes of dissimination including journals and presentations."

This was the only comment this reviewer left for the broader impacts category. In my opinion you can't really beat the type of outreach where you are literally going house to house doing research in a person's backyard and using the opportunity to educate them on science that is taking place in their own backyard about the wildlife that lives in it. Projects such as the Smithsonian's Project Nest Watch and Cornell's Project Feeder Watch use similar methods and I drew from these citizen science programs. In addition to this outreach I also needed help collecting certain samples and suggested that landowner's would be important for collecting certain data. This made a portion of my proposal interactive with the community.

Here are my questions.

Do you think I am overreacting to this? I am looking to see if other people have had similar experiences. Is this just the game when you have over 16000 applicants?

While I can't appeal (even if I did, I don't think I got a good enough review from another reviewer) is there a way to provide feedback on a reviewer? I feel like my proposal wasn't given the thought and consideration deserved considering how much work was put in. I was hoping that maybe there is a process where if several applicants had similar experiences and gave feedback perhaps this reviewer would not be a reviewer for future applicants.

My lab mates and friends are on my side and trying to be supportive, but I want some outside opinions since they are obviously trying to be consoling and thus I am not sure if my issues are even valid. Thanks for any thoughts on the matter!

Edited April 2, 2015 by mheimbu2

[Aliake]

1st time poster here. I had been following this thread and wanted to ask a couple questions to fellow GRFP applicants. I should start by saying that that I was not recommended for a fellowship and this was my last and only attempt. I know there is no appealing the decision, and I am not really trying to, but I wanted to hear from some other people on the feedback they received and maybe get some advice.

While I was obviously disappointed that I did not receive an award I was pretty upset about one of the reviewers comments that strongly implied they had barely read my application. Here's some context.

My proposal was in the field of ecology and the project involved urban ecology specifically. Due to the nature and design of the project I would need to almost exclusively use private lands in my work. I proposed that I would gain access to private lands by speaking to local Audubon Societies (my project involved birds), conservation organizations, and neighborhood organizations, hoping to generate interest and get landowner's to volunteer me access to their property. These properties would be more like backyards in a suburban landscape. In my proposal I suggested that working in people's backyards would allow me to do outreach and education. This outreach and would be basically house-to-house and one-on-one by necessity. This is the feedback on my broader impacts, they gave me a "fair" assessment.

"The broader impacts included are traditional modes of dissimination including journals and presentations."

This was the only comment this reviewer left for the broader impacts category. In my opinion you can't really beat the type of outreach where you are literally going house to house doing research in a person's backyard and using the opportunity to educate them on science that is taking place in their own backyard about the wildlife that lives in it. Projects such as the Smithsonian's Project Nest Watch and Cornell's Project Feeder Watch use similar methods and I drew from these citizen science programs. In addition to this outreach I also needed help collecting certain samples and suggested that landowner's would be important for collecting certain data. This made a portion of my proposal interactive with the community.

Here are my questions.

Do you think I am overreacting to this? I am looking to see if other people have had similar experiences. Is this just the game when you have over 16000 applicants?

While I can't appeal (even if I did, I don't think I got a good enough review from another reviewer) is there a way to provide feedback on a reviewer? I feel like my proposal wasn't given the thought and consideration deserved considering how much work was put in. I was hoping that maybe there is a process where if several applicants had similar experiences and gave feedback perhaps this reviewer would not be a reviewer for future applicants.

My lab mates and friends are on my side and trying to be supportive, but I want some outside opinions since they are obviously trying to be consoling and thus I am not sure if my issues are even valid. Thanks for any thoughts on the matter!

 

I can certainly understand your frustration. While I cannot comment on the existence of an appeal process or applicant feedback system, I can say that I don't think you are overacting. The only defense I can think of for the reviewer is the fact that he may have assessed your broader impacts based on your holistic academic career thus far. In fact, I believe two of my reviewers did just that. Their comments focused on the fact that I was a volunteer at the children's museum in town and was a student mentor for summer research programs on campus, They barely touched on what I had said in my actual proposal. This is obviously confusing, and unfortunately I think there's just a human error element regarding the way reviewers interpret the criteria. Some place much greater emphasis on the proposal than the rest of the app and vice versa. Now, if you've also been involved in outreach, etc. then there's absolutely no good excuse for such a short and irrelevant comment.

[Eigen]

I wouldn't say you're overreacting, but I would say this is a very good introduction to getting scientific research funded. 

 

My initial thought would be to take away the fact that your proposal was not clear enough on the broader impacts relative to the time the reviewers have to read it. 

 

Did you use underlining, bold, and specific introductions like "The broader impact of this study is XXY", or say that it matched up with the broader impacts from Project Nest & Project Feeder?

 

Getting funding in the sciences is a confound of several issues: How good your idea is, how popular your idea is, and how effectively that idea is communicated. There's also a nice topping of "how cranky the reviewer was on the morning the read your proposal", but there's nothing you can do about that. Funding (including the GRFP) is an organized crapshoot, and there's only so much you can do to mitigate the effects of the review process. 

 

You feel like you weren't given enough thought and consideration given the time put in, but that's part of this. Reviews of a whole dossier take minutes. The same will be true for grant funding reviews, as well as peer review of articles you write. 

 

For coping, I suggest venting, being frustrated, putting the review aside for a month or two, and then coming back to it. When you come back to it, approach it productively: What, if anything, can you do to mitigate this happening to you again. Is there any merit to the review, or any of their suggestions. Take what you can away from it, make changes as appropriate, and move on. 

 

I've had to deal with this from peer reviews of articles- reviewers rejecting manuscripts based on the fact that I "missed referencing essential work from Smith et al", when I devoted the better part of a page to the discussion of Smith et al. Sometimes you can write a response, sometimes you just have to eat it and move on.

[Argon]

Based on recent comments, I do also wanted to chime in and say that I do believe NSF cares more about the broader impact of the applicant in the statement of purpose than the proposal. My research is on finding a greener diesel fuel, and I had an entire half-page dedicated to broader impacts and titled it as such. However, the reason I didn't get the fellowship was because two reviewers gave me a G for broader impact (the third gave me an E). They both mentioned nothing about the broader impact of my research and only about the relative lack of broader impact in me as a person. I don't go out and help people and volunteer and do all that, so it's justified. I had a paragraph on broader impact in my statement of purpose, but apparently that wasn't enough. The third reviewer who gave me an E seemed to love the broader impact of my research. So, it seems the NSF cares more about the broader impact of you as an applicant than your work. For me, I am my work, and so that hurt my chances. Nonetheless, for people that got off for broader impact but had proposals that literally helped the people, it may not have mattered because NSF may focus more on the personal statement for that. 

[Eigen]

Just FYI, under the NSF review criteria (not just for the GRFP),broader impacts are primarily outreach. Here's the most recent "dear colleague" letter detailing broader impacts:

 

 

Advance discovery and understanding while promoting teaching, training, and learning, for example, by training graduate students, mentoring postdoctoral researchers and junior faculty,  involving undergraduates in research experiences, and participating in the recruitment, training, and professional development of K-12 mathematics and science teachers.

Broaden participation of under-represented groups, for example,by establishing collaborations with students and faculty from institutions and organizations serving women, minorities, and other groups under-represented in the mathematical sciences.

Enhance infrastructure for research and education, for example, byestablishing collaborations with researchers in industry and government laboratories, developing partnerships with international academic institutions and organizations, and building networks of U.S. colleges and universities.

Broaden dissemination to enhance scientific and technological understanding, for example, bypresenting results of research and education projects in formats useful to students, scientists and engineers, members of Congress, teachers, and the general public.

Benefits to society mayoccur, for example, when results of research and education projects are applied to other fields of science and technology to create startup companies, to improve commercial technology, to inform public policy, and to enhance national security.

 

For grant writing, GRFP among them, you really want to be sure you know what the review team is looking for, and give it to them. See the above bolded phrases? You'd want to use language as similar to that as possible to talk about what you're doing. Look for keywords in the criteria, use those keywords to describe your research/broader impacts.

 

Heck, even bold the sections where you use those keywords.

 

Do the same thing for intellectual merit.

 

Panels reviewing the GRFP are usually also those that review other NSF grants. I also personally recommend NSF's proposal writing guide (http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/gpg_2.jsp#IIC2d). Not all of it is applicable, but it has a lot of fantastic information in it that helps you with grant writing in general and NSF criteria in particular.