NSF GRFP 2011-2012

[AntClimbsTree]

Can anyone comment on the following?

IM/BI (CS Major, 2nd year)

E/E

E/VG

E/E

HM only no award.

Reviews were all positive and no negatives (snippets below):

The problem is important and clearly explained; the

challenges are articulated. The applicant has demonstrated a strong record of success.

The proposed study could have broader impacts in many areas. The technology can enhance scientic

and technical understanding of oceans.

He has an extensive research experience. He has an outstanding record of published papers. He has demon-

strated intellectual ability to plan and conduct research.

The applicant has demonstrated leadership skills.

The applicant has ambitious plans to continue diversity and outreach work.

Not sure how I didn't win but its my last time eligible

Has anyone ever heard about appealing the results and winning?

[Jana Olson]

E/E

E/E

VG/G

awarded

My essays focused on my writing and communication skills, which my resume and letter writers backed up. I pointed out the low level of scientific literacy in the public, and how my love of communicating science to others (and my background - I do a bit of volunteering) makes me an excellent candidate for raising the scientific literacy. Basically, I took a risk or two with some bold statements in my personal statement, but it worked out in the end.

Also, I think it's one thing to cover all the bases, but quite an other to really stand out in writing. My proposal sucked last year (got nothing), and this was my last chance.

Edited April 3, 2012 by Jana Olson

[vertices]

Can anyone comment on the following?

IM/BI (CS Major, 2nd year)

E/E

E/VG

E/E

HM only no award.

Reviews were all positive and no negatives (snippets below):

<snip>

Not sure how I didn't win but its my last time eligible

Has anyone ever heard about appealing the results and winning?

That's rough AntClimbsTree. It sounds like z-scoring worked against you here. I haven't heard of appealing.

It's hard to say anything definite about your snippets without your essays. I'm going to assume they're direct quotes. While they are positive, there's a difference in specificity and strength among them:

The proposed study could have broader impacts in many areas. The technology can enhance scientic

and technical understanding of oceans.

Note the use of 'could' and 'can' here. In other review sheets I've read, these words are sometimes used when the applicant did not be specific, explicit, or lengthy enough with the proposal BIs. These sentences sound like the reviewer is trying to come up with the BIs for you.

The problem is important and clearly explained; the

challenges are articulated.

He has demon-

strated intellectual ability to plan and conduct research.

The applicant has demonstrated leadership skills.

These sentences are written like the reviewer is checking off something. "Explained challenges -- check. Leadership skills -- check." There's no emotion in them.

The applicant has ambitious plans to continue diversity and outreach work.

The use of 'ambitious' here is better... but it could also indicate the writer has doubts to feasibility. I'm not sure.

The applicant has demonstrated a strong record of success.

He has an extensive research experience. He has an outstanding record of published papers.

Here we see 'strong', 'extensive' and best of all 'outstanding'. There will be many applicants with 'strong' records of success, but 'outstanding' in this group really says a lot.

That's my analysis of how positive reviews can still have room for improvement. Of course, this is just my guess on the matter. It could be that the reviewers you had just don't write as effusively as others. Still, when the reviewer takes time to write in adjectives, you get more of a feeling that s/he is more likely to have given you closer to 50 than 40 for an E and more willing to fight for your application during re-ranking.

[AntClimbsTree]

I did not want to post the entire review but I guess I should have because without the context some comments could go either way. I also spoke to a former reviewer about it and they said there isnt enough time to provide "detailed comments" but that it looks as if I should have received an award. Not sure what to say though.

Regardless, I still feel the result should have ended up differently, too subjective in my opinion.

[maath805]

Got nothing. G/G G/F

[bemonocled]

Got an honorable mention for math, with VG/VG, G/G, VG/E.

[crimsonengineer87]

I found this Reviewer's Guide to be the most informative in regards to the scoring process (at least as it was in 2008):

http://www.soest.haw...fo_NSF_GRFP.pdf

**thanks to vertices for posting it earlier in the thread**

Starting on page 11, it shows the numerical rating scale behind the E,VG,G,F,P ratings that we applicants get to see.

It also explains the standardized scoring (z-scores) that is supposed to combat the effects of varying reviewer difficulty (pg 13).

According to this guide, applicants were divided into 4 "quality groups:"

Applicants in Group 1 are all awarded fellowships

Applicants in Group 2 "receive awards to the limit of funds available using criteria such as geographical region, discipline, and other factors" (pg 16); the rest receive HMs

Applicants in Group 3 get HMs

Applicants in Group 4 do not get awards or HMs (this group includes applications that were below the 65th percentile after two ratings and were retired before the third rating).

From what I read it seems, in 2008, applicants were ranked and placed into the quality groups according to their z-score averages, but the panels of reviewers were able to deliberate and change ranks without changing scores (pg 14). So there is definitely some mystery surrounding the deliberation aspect of the rankings.

Plus, once the rankings are submitted, who knows what "other factors" the NSF uses to give additional awards to applicants in Group 2.

I think the combination of z-scores (which applicants don't see) and the reviewing panel's ability to change the rankings after scoring helps explain why the feedback we get (just the letter ratings and some comments) is often not helpful in determining why we did/didn't get an award/HM.

Edit: Jimbo's comment above coincides with this as well; I agree it would be nice to get at least the z-scores.

Does someone have the actual pdf they can send me or repost somewhere else? The link doesn't work. I'd also like to know more about the scoring process

This year, I got

E/G

E/E

E/VG

I applied under Geosciences and my research involves modeling of nitrogen in agriculture. I completely agree that the process is arbitrary! Compared to the reviews from last year, the ones from this year were horribly written (grammar-wise, spelling, etc) and kind of looked like they just stated what I had successfully accomplished off a rubric. While I thought getting a Fulbright, beefing up my BI, and re-tooling my research statement would be reasons for potentially getting GRF, I think it was more that I clearly organized my statements moreso and just got different reviewers. Who knows ...

[TruBeliever]

I ended up getting the NSF Fellowship .

I am in the chemistry division.

VG/E

E/E

EG/G

I am surprised but I'll take it. I have a question though. I also applied for the NDSEG and DOE and haven't heard yet. If I happen to get another one what should be my best option. Which one is more prestigious and the like?

[zero_percent_agarose]

I have a question though. I also applied for the NDSEG and DOE and haven't heard yet. If I happen to get another one what should be my best option. Which one is more prestigious and the like?

If you measure prestige in selectiveness and amount of money then NDSEG is the "better" award.

NDSEG

~200 awards, ~10% admit rate (http://ndseg.asee.or...the_application), and actually, for your field (Chemistry) the admit rate looks to be more like 6% (http://ndseg.asee.or...award_selection)

Tuition: Full for 3 years

Stipend: $30,500+$31,000+$31,500

NSF

~2000 awards, ~15% admit rate (www.wpi.edu/academics/FS/nsf.pdf)

Tuition: $12,000 tuition/yr for 3 years

Stipend: $30,000/yr stipend for 3 years

Obviously, it's an honor to get either, so congratulations!

EDIT:

This post from last year has a better-explained breakdown of the differences between the two. Bottom line: it'll depend on the funding structure of your program, your particular field of study, and your own preferences.

Edited April 4, 2012 by zero_percent_agarose

[mcguirbc]

My reviews varied so much:

Mathematics-Statistics

E/F

VG/E

G/VG.

Nothing....

[curiousgeorgie]

So here I was, feeling like I had a decent shot at this thing. I had a pretty stellar undergrad GPA/GRE, a bunch of research experience (though no publications because work was still going on for that paper and it's still being written now), strong letters of rec, the resources of a top school in my field, lots of volunteer/outreach experience, and a pretty good writing style. So when I didn't get it, I thought, oh well, there's tons of qualified applicants, I'll just work on whatever the reviews say for next year (which will be my last chance).

G/G, VG/VG - I was fine until I realized there were supposed to be 3 reviewers and I got dropped before I even got to a 3rd. Well crap, I didn't think I was that bad...

Yes, I know that it depends on which reviewers you get, the fact that I don't yet have any publications (mentioned explicitly by the first reviewer), and how well you play their game. That's what I keep telling myself, but it's not really working right now. I'll get over it.

Anyways, I thought I devoted plenty of time to broader impacts, but next year, it's just going to have to be completely over the top, and I'm going to make it as obvious as possible what those broader impacts are.

For anyone who didn't have helpful reviewers and is curious about what kinds of things they might need to work on with broader impacts, I can at least mention some of the constructive criticism I got and hope it will be useful:

This applicant has engaged in several outreach activities to bring science education to K-12 students in both low-income urban areas and very poor rural areas. A more detailed description of other future plans (broader impacts ) would strengthen this application.

The applicant has thought a lot about broader impacts and uses her own background and experience to make a difference. This is very important. If you need to reapply spend a bit more time talking about how you will connect your academic career with your drive for broader impact activities.

And, even in the intellectual merit section:

The proposal is well written. Possible alternate results and ways to deal with them could have been discussed. The implications of the work and broader

impacts not well mentioned.

Congratulations to all those who got the award!

[firefly luciferase]

@curiousgeorgie, did you at least include the paper as a "manuscript in preparation"? I would think that would give you a few brownie points.

[vertices]

I'm trying to compile a list of things I've seen reviewers comment on. I figure this would be good for the 'what's missing' elements when reviewers only write positives. (Though note, there is a wide range of how positive a comment is.) Here's what I have so far in no particular order. Others?

IM

- whether proposal has a clear hypothesis/question it is trying to address

- whether proposal is novel/innovative

- whether proposal is transformative (rare)

- whether proposal is well-written

- whether access to resources is addressed in proposal

- whether expected results are addressed in proposal

- whether validation of results are addressed in proposal

- whether proposal is sufficiently detailed (this is a tough one... too detailed and it might not be well-written/clear and have no room for other objectives)

- interest of proposal among scientists

- strength academic record

- publications

- presentations

- research capabilities, independent and in team

- leadership potential of applicant

- strength of recommendation letters and what they say about all of the the above

BI

- whether proposal integrates science education with research

- whether proposal integrates broadening diversity with research

- whether proposal benefits society or some large group outside major field

- applicant's plan to disseminate results broadly, especially with non-science stakeholders if applicable, but further too

- applicant's activities to increase science education and recruit young scientists

- applicant's activities to broaden diversity in science

- applicant's leadership roles in broader impacts activities

- quantity of applicant's BI activites (e.g. reviewer lists lots of examples or length of time)

- applicant's plan to -continue- broader impacts activities

- specifics for BI activities

Many thanks to all of those who have posted reviewer comments or reviewer comment summaries. I understand some people don't want to post whole reviews due to privacy issues (at least that's why I don't), so hopefully these summaries can help those who can still apply.

The most common suggestion I see is the need to integrate BI activities (benefit to society, dissemination of results, diversity, education/outreach) in the proposal. Often times someone will have demonstrated such activities in their personal statement, but be told they haven't done enough in their proposal... probably because there's so little room!

Maybe we should start a wiki to go along with that essay document store people were proposing up thread.

[firefly luciferase]

Great post, vertices. A few additions:

IM

- potential/preparedness to carry out proposed research

- possible pitfalls and alternatives in proposal

- "proposed graduate institution"/department is appropriate to proposed research (i.e. proposal aligns somewhat with work being done by professors in the department)

BI

- long term career goals

[luce373]

E/VG

E/VG

E/E

Offered Award

I think that advice from former winners played a big role in my results. The things that people told me to emphasize ended up getting highlighted in my reviews as positives.

IM:

-I came up with the project on my own (with some guidance from my PI, of course), and he mentioned this in his letter.

-The project was a fusion of work done in my undergrad and grad labs, and the reviewers thought this was sophisticated and showed good precedence for the work. (I have heard this can go either way, though...being too tied to your undergrad projects can make you seem like you're not creative enough.)

-Two different reviewers mentioned my persistence and drive, which I think was highlighted most in my letters

BI:

-My broader impacts were really, really thin so I overemphasized what I had. I have done maybe three or four outreach days in my life between high school, college, and grad school but I described them in enough detail it was interpreted as a long-term commitment to outreach. The way you frame things is so important.

-They also liked that I was a TA and I discussed how that reflected both leadership and communication/education. I also discussed the leadership and teamwork I had in my lab even though there were no official positions.

-One reviewer also noted the conferences I attended in this category because it showed scientific communication (which I said explicitly in the essay). So even things that seem like IM criteria can be twisted for BI.

[firefly luciferase]

One more for BI:

Demonstrating a broader range of interests outside of science/engineering can be a good thing, as long as taking the time to discuss these doesn't detract from your scientific BIs. For example, a small part of my personal statement described how I had volunteered/traveled extensively through several non-technical service organizations as an undergrad, and the reviewers responded positively to this. Another winning essay/review I read included having minored in some type of international/cultural studies in undergrad.

[hangin brain]

I'll add one BI comment that I got from all 3 of my reviewers:

- "unique perspective" due to unusual life circumstances which gives applicant insight into the needs of underserved communities.

If you've ever been through some weird shit, weave it into your personal statement. All 3 of my reviewers this year, and both of them last year, commented positively on that aspect of my application.

[Pitangus]

Here are some comments I got that I thought were interesting/different.

For IM:

- Reviewer gave me "kudos!" for a sole-authored publication, even though it was only submitted.

- About "attention to detail" in my previous research essay: reviewer thought it was "quite helpful" that I explained "how each research experience contributed to [my] development as a scientist."

For BI:

- Reviewer thought the citizen science aspect of my proposal was a "potentially exciting" way to communicate science to a broad audience.

[Dynamom]

GREAT IDEA

Hope one of us remembers to re-post this in next year's thread. I added the other posts from this page plus my own...

IM

- whether proposal has a clear hypothesis/question it is trying to address

- whether proposal is novel/innovative

- whether proposal is original (your own)

- whether proposal is transformative (rare)

- whether proposal is well-written

- whether access to resources is addressed in proposal

- whether expected results are addressed in proposal

- whether validation of results are addressed in proposal

- whether proposal is sufficiently detailed (this is a tough one... too detailed and it might not be well-written/clear and have no room for other objectives)

- interest of proposal among scientists

- strength academic record

- publications

- presentations

- research capabilities, independent and in team

- leadership potential of applicant, including personal qualities such as persistence and drive

- strength of recommendation letters and what they say about all of the the above

- relation of research to theories of founders/experts in the field

- whether the outcome measure is well-related or "distal" to the intervention (likely more of an issue in social sciences)

BI

- whether proposal integrates/supports science education with research

- whether proposal integrates broadening diversity with research

- whether proposal benefits society or some large group outside major field

- potential of research to support "citizen science"

- applicant's plan to disseminate results broadly, especially with non-science stakeholders if applicable, but further too

- applicant's activities to increase science education and recruit young scientists

- applicant's activities to broaden diversity in science

- applicant's leadership roles in broader impacts activities (including TAing)

- quantity of applicant's BI activites (e.g. reviewer lists lots of examples or length of time)

- applicant's plan to -continue- broader impacts activities

- specifics for BI activities

- "unique perspective" due to unusual life circumstances which gives applicant insight into the needs of underserved communities

Edited April 5, 2012 by Dynamom

[Pitangus]

I agree with Dynamom. I've added my feedback about publications and the previous research essay.

Perhaps when it is time to re-post for next year we can arrange the list into sub-headings ("proposal" etc.) to make it more manageable.

IM

- whether proposal has a clear hypothesis/question it is trying to address

- whether proposal is novel/innovative

- whether proposal is original (your own)

- whether proposal is transformative (rare)

- whether proposal is well-written

- whether access to resources is addressed in proposal

- whether expected results are addressed in proposal

- whether validation of results are addressed in proposal

- whether proposal is sufficiently detailed (this is a tough one... too detailed and it might not be well-written/clear and have no room for other objectives)

- interest of proposal among scientists

- strength academic record

- publications (includes submitted/in review articles, especially if you don't have other pubs yet)

- presentations

- research capabilities, independent and in team

- leadership potential of applicant, including personal qualities such as persistence and drive

- strength of recommendation letters and what they say about all of the the above

- relation of research to theories of founders/experts in the field

- whether the outcome measure is well-related or "distal" to the intervention (likely more of an issue in social sciences)

- previous research essay draws connections among past projects and internships, and explains their significance (both personal + BIs)

BI

- whether proposal integrates/supports science education with research

- whether proposal integrates broadening diversity with research

- whether proposal benefits society or some large group outside major field

- potential of research to support "citizen science"

- applicant's plan to disseminate results broadly, especially with non-science stakeholders if applicable, but further too

- applicant's activities to increase science education and recruit young scientists

- applicant's activities to broaden diversity in science

- applicant's leadership roles in broader impacts activities (including TAing)

- quantity of applicant's BI activites (e.g. reviewer lists lots of examples or length of time)

- applicant's plan to -continue- broader impacts activities

- specifics for BI activities

- "unique perspective" due to unusual life circumstances which gives applicant insight into the needs of underserved communities

Edited April 5, 2012 by Pitangus

[curiousgeorgie]

@ firefly luciferase: At the time, I couldn't think of a way to say that concisely, and I ended up forgetting in the midst of all the other things I was trying to think about. If it's not yet out for review or in line for publication by the time I apply again this fall, I will definitely say that, thank you!

These compilations of what reviewers have mentioned are going to be extremely useful in thinking about how I need to rework my essays for next year, so great idea!

[vertices]

Awesome additions guys! I've added a few and incorporated Pitangus' suggestion to divide these up for easier reading. I kept the personal statement and research experience together because there's some overlap in how you want to handle them.

INTELLECTUAL MERIT

- PROPOSAL

-- whether proposal has a clear hypothesis/question it is trying to address

-- whether proposal is novel/innovative

-- whether proposal is original (your own)

-- whether proposal is transformative (rare)

-- whether proposal is well-written

-- whether access to resources is addressed in proposal

-- whether expected results are addressed in proposal

-- whether validation of results are addressed in proposal

-- whether proposal is sufficiently detailed (this is a tough one... too detailed and it might not be well-written/clear and have no room for other objectives)

-- interest of proposal among scientists

-- relation of research to theories of founders/experts in the field

-- relation to previous work the applicant discusses in other essays/supporting material

-- whether the outcome measure is well-related or "distal" to the intervention (likely more of an issue in social sciences)

- PERSONAL STATEMENT/RESEARCH EXPERIENCE/SUPPORTING MATERIAL

-- strength of academic record

-- research capabilities, independent and in team

-- publications (includes submitted/in review articles, especially if you don't have other pubs yet)

-- presentations (includes talks and posters)

-- participating in any other funded proposals (e.g. help writing grants)

-- leadership potential of applicant, including personal qualities such as persistence and drive

-- previous research essay draws connections among past projects and internships, and explains their significance (both personal + BIs)

- ALL

* strength of recommendation letters and what they say about all of the the above (including proposal if applicable)

BROADER IMPACTS

- PROPOSAL

-- whether proposal integrates/supports science education with research

-- whether proposal integrates broadening diversity with research

-- whether proposal benefits society or some large group outside major field

-- potential of research to support "citizen science"

-- applicant's plan to disseminate results broadly, especially with non-science stakeholders if applicable, but further too

- PERSONAL STATEMENT/RESEARCH EXPERIENCE/SUPPORTING MATERIAL

-- benefits to society, education or diversity in all past research projects

-- applicant's activities to increase science education and recruit young scientists

-- applicant's activities to broaden diversity in science

-- applicant's leadership roles in broader impacts activities (including TAing)

-- quantity of applicant's BI activites (e.g. reviewer lists lots of examples or length of time)

-- applicant's plan to -continue- broader impacts activities

-- "unique perspective" due to unusual life circumstances which gives applicant insight into the needs of underserved communities

- ALL

* specifics for BI activities in all sources

* strength of recommendation letters and what they say about all of the the above

@Dynamom: Maybe next year's thread's kickoff should be a group mining expedition of all the previous threads. I read them all at some point (including all 170+ pages of the year-less one) and incorporated trends but didn't write the information down back in November when I probably should have.

[AnnzPB]

I'm sure all you applicants have seen this document already (or perhaps not, who knows), but I went through this Broader Impacts Activities document that NSF published to get a very clear sense for the types of things that they consider under Broader Impact, and I was awarded this year as a pre-graduate school student. Like most winners have already mentioned, the more specific you are in what you're going to do IN THE FUTURE (and tying that to your previous activities), the better.

http://www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf

[vertices]

I agree that the rating sheets are meant to be helpful for future applications/proposals, and they can be in cases where the comments provide constructive criticisms or suggestions.

I'm just not seeing the usefulness of including the E/VG etc scores. If the point is to give applicants an idea of how well they did in terms of the scoring procedure, why not use the numerical scale if that's what reviewers really use?

Yes, those scores still aren't that useful for improving an application, but maybe they'd cause less confusion/frustration.

I like your idea of reviewers adding points rather than detracting based on some idea of "best."

It makes sense considering the way most reviewers seem to be commenting (mostly positives), though it requires applicants to think backwards, looking not just for aspects that were criticized but also ones that weren't praised (but perhaps did receive praise in other people's applications).

I think the numerical scale would have some of the same problems E/VG does -- it would still confuse people to have higher numbers than others who won. Even if z-scores were given, we'd have the same confusion due to 1) re-ranking and 2) differences in winning z-scores across panels -- even in the same major field of study since many of them have multiple panels. This leaves the actual rankings within the panel groups, but there's a problem there too: geographical bonuses that come into play AFTER re-ranking if you're in Q2.

I think what might be useful is a modified, coarser grain releasing of the panel rankings:

1) Don't give out ranks to those offered awards. Yes, this means those offered awards don't get as much information, but I don't think it's a good idea to point out who likely got an award due to geography. Furthermore, it might even lead to things like encouraging people to list their rank if they won (since high ranking people might) or high ranking HMs comparing themselves to winners.

2) Divide the remaining applicants into rank-deciles, covering the whole range of remaining ranks. Fudge this a little so that the Q3 and Q4 don't overlap a decile (because having an HM and a non-HM in the same decile might cause problems). Let them know their deciles "e.g. Top 30% of non-awardees." Since panels review 250-300 applications, assuming ~15% get awards, this leaves at least 200 applications, so the deciles would cover about 20 people each. (At this point they could also just take the remaining 200 and renumber them 1-200, but I think fine-grained rankings like that would be misleading because the process to come up with ranks isn't that precise.)

3) Give everyone their reviewer comments of course!

4) I think everyone (award or no) should also get their public K-12 textbook scores (E/VG/G/F/P) too. It's something closer to what the individual reviewer is thinking and I think that's important for everyone's future proposals. I think having positive comments with an E and positive comments with a G do tell us something, even if we're not quite sure what. Reviewers care a lot about science and students and are working hard all weekend to get through the applications while trying to be fair, but they aren't infallible and time is short -- that's why we're now mining comments to make more sense of the feedback, but this is something that's going to continue to vex us for the rest of our publishing and grant writing lives. The decile-rankings and the E/VG ratings will tell different parts of the whole story to the non-awardees.

[Pitangus]

Didn't people share percentile ranges in previous threads? 2009 maybe. They were pretty broad ranges though, and I don't know what they actually represented.

The NSF must have stopped reporting them (or started hiding them better).

Perhaps the percentiles reflected the Quality Groups to some degree, and like you said, the NSF didn't want winners to figure out whether they were in Group 1 or 2, because of the geographical factor that seems to account for at least the first round of awardees from G2. (Though I think people can make assumptions in some cases even without the percentiles/rankings; for example, I would be surprised if my state was not either well- or over-represented in most panels).

But this is all speculation. Clearly the NSF just doesn't want to give out the numerical scores/rankings/percentiles, for whatever reason.

The E/VG scores are ok for individuals, but these threads show that once you start comparing people with Es/VGs/Gs, it just becomes unhelpful and confusing.

Edit: To crimsonengineer and anyone else who wanted the 2008 Reviewer's Guide, it does look like that link has been removed. Sorry about that. I will see if I saved it at any point; has anyone else seen one like it posted elsewhere?

Or maybe it's a sign that we need to stop thinking about this. Fair enough

Edited April 5, 2012 by Pitangus