NSF GRFP 2010-2011

[BlueRose]

Let's quit beating the poor dead thread from last year (173 pages?!?) and start a new one. Who's with me?

I'll start the proceedings with a collection of links. First, I have one of the best descriptions of the NSF process that I have found. It's a bit dated (1991) and written in the style / jargon of an anthropology journal article, but if you can get past that, there's some good insight on how they make decisions.

Discourse and Discipline at the National Research Council: A Bureaucratic Bildungsroman

Next, I have a sample essay that...well, I'm fairly sure it wasn't funded. But you might like it anyway.

Harnessing Geothermal Energy for the Eradication of the Energy Crisis, Global Warming, and Zombies while Defeating Social Injustice in Education and Providing Free Heath Care to All Americans

Finally, I have posted my own essays for posterity. I think they're pretty good...but we shall see. As the Demotivators poster says, perhaps their purpose in life is to serve as a warning to others.

Simulated Evolution for Synthetic Biology: A Case Study on Biological Counters

Enjoy. And then quit refreshing Fastlane every ten minutes...it's not April yet!

[adaptations]

I'll join the fun. I applied in the social sciences. I've had so many friends go through this process, none who won the award, many with honorable mention, and yet I have almost no idea what to expect.

I started reading the first article you recommended - the beginning is a like a barbwire fence discouraging anyone from reading further.

[BlueRose]

I can excerpt the first article, if it helps. (My tolerance is fairly high - I came within a course of majoring in anthropology.)

The good part starts on p. 26, "Into the Panopticon: Application Reading at the NRC". The author served on one of the Social Science panels, and describes the mechanics. On the statistics:

"In the three years in which I was involved, anthropology, archaeology, sociology, linguistics, and "social studies" constituted the Social Sciences B (Soc- SciB) panel. There were between 18 and 20 of us on the panel, our disciplinary representation roughly proportional to the distribution of applicants' fields. In 1991...510 applications were assigned to SocSciB."

Then there's some stuff about a derived score, which I don't think is still in effect; now everyone gets at least two readers before the cut is made. There is still a cut, however; if you're below some cutoff after two reviewers (<70%?) then you don't get a third.

Then it discusses how the results are tweaked before being finalized:

"The computer also notes all cases where there is a difference of more than 1.5 points (on the 6-point scale) among panelists for any particular folder. We are told that the computer will not accept as final any score with that degree of disagreement. NRC's concern here is not panelist amity. It derives from the underlying assumption that, as an objective phenomenon, applicant quality should be represented by more closely clustered scores. The outlying readers have to negotiate their differences and come up with a final score that can be accepted as sufficiently close to consensual. Only rarely are these negotiations lengthy, and even less frequently are they contentious. It is also critical to note that, while there has been a great deal of sociable talk around the table throughout the preceding days, only rarely have applicants been discussed. Only on the last morning and only when our aggregate evaluations are signaled as impermissible, do we talk directly about particular cases. The computer not only makes critical cuts in the applicant pool along the way and provides us with information for self-monitoring; it also tells us when to get down to brass tacks.

Our final task is to make fine adjustments in the ranked list of candidates. While only members of Quality Groups I and II are eligible for awards, there are six quality groups altogether. Those candidates in QGII who do not receive awards and all in QGIII will receive honorable mentions. The 35 candidates with the highest average scores constitute QGI; they are guaranteed awards. Approximately half of the 42 applicants in QGII (the next cluster of averaged scores) will be Fellows, although the decision is not in our hands but in those of NSF. Here panelists' anticipation of those criteria that NSF will use in making these decisions becomes significant. Particularly for those of us on SocSciB, a concern that our QGII candidates stand up well vis-a-vis applicants in the sciences was critical. It led at times to slight adjustments at the lower end of the Quality Group II ranking to favor those applicants with somewhat better quantitative profiles. We anticipate our subsequent audience in determining the final membership of QGII."

Then it goes into the process for choosing the people in QGII who will receive awards:

"The computer first surveys all the QGI members to determine if each state is represented among the awardees. If, for example, no one who has graduated from high school in Alabama is in QGI and one is available in QGII, he or she will be chosen. ... If the state search does not fill the available slots, a second pass is done for gender balance, followed by disciplinary balance and proportional equivalence between [pre-grad school] and [current grad student] applicants."

Afterwards, there is a section on what this means:

"the number of proposals and the expected degree of attention and response to them always come close to exceeding readers' capacities...frequently we are working close to the limits of wit and energy."

"The certainty of a subsequent and powerful audience has a real effect on one's conduct in the panel: What do you think will fly with later readers? What might not? How much of your panel-member credit will you expend on the unlikely candidate, however intriguing? And how many times can you go out on a limb?"

"a concern for fairness and something like due process in treating the proposals...places such textual aspects of the proposals themselves as clarity and comparability at a premium..."

"The highly innovative project or, more particularly, the proposal that changes the terms of research formulation and design, is often, even if engaging and provocative, difficult to fit into such a common comparative language. Here lies one of the costs of what I think of as the "fairness-through-apparent-clarity" model of proposal review. This apparent clarity approach is inherently biased toward an understanding of scholarly progress as incremental and therefore often leads to the favoring of more or less "normal" social-science features that are clearly linked to a sense of how "science" works. ... Proposals that promise to break new conceptual ground... are viewed not so much as "bad" proposals but as difficult to evaluate and compare with other contenders."

"in each panel, some means of monitoring and calibrating one's conduct is provided. ...our initial conversational partner is a computer-generated score for each applicant, the derived score. ... The score sheet further allows us to anticipate, think about, and revise our own evaluations before getting to the discussion of particular cases."

"feedback makes self-monitoring not only possible but almost inescapable. ... Few participants are comfortable as consistent outliers, those who are repeatedly more positive or negative than the middling evaluation. A set of expectations as to socially feasible disagreement... are central to panel dynamics, as well as to the criteria by which program officers select those panelists who will be invited to serve again."

Basically, this is why you have to explicitly address all the criteria on the score sheet, even if you think you are totes awesome at one or two of the things and should therefore get full credit. They are trying to compare hundreds of proposals "fairly", and have to mechanically go down the list of things they grade on.

Edited December 20, 2010 by BlueRose

[adaptations]

I am intrigued by the "state" criteria. If they use the state where you graduated high school, it inherently biases applicants from states with a lower population. So if I come from North Dakota there is much less of a chance that anyone else from North Dakota will be in group I or II, meaning that I am more likely to get pulled up from group II to I than if I were an applicant from California or New York, where there is most likely at least one more competitive applicant. Granted, you still have to make it into the top two groups, but it is a peculiar criteria.

[Krypton]

Yep, since NSF is federally funded, balanced state representation among fellowship recipients is government-mandated.

[t_ruth]

I'm in the waiting game as well. Applied under STEM Ed., Math. Waiting to do all my refreshing until late March...maybe there will be another glitch/slip-up this year and things will come early

[t_ruth]

Thank you BlueRose for sharing those. What do you think about the prereading quantitative formulas? Since they are not considering GRE score this year, what do you think will happen to that? I have to admit, I'm pretty disappointed, as I had a really good GRE for my field, but graduated (almost 15 years ago!) from a second-rate undergrad school w/a fairly mediocre GPA. My worry is that I won't make it past this round.

[BlueRose]

What do you think about the prereading quantitative formulas? Since they are not considering GRE score this year, what do you think will happen to that?

I think they've phased out using an explicit pre-reading score - maybe they still give it to panelists, but I know they don't use it to make cuts. The GRE has been optional for years, but I was still surprised that they cut it entirely. I'm another one whose GRE is significantly better than my undergrad record...sigh. At least it meant I didn't have to take the Subject GRE.

As for the state-based selection, I don't think it's as big of a deal as people make it out to be. As I understand it, this only comes into play if nobody from a state has made the top 7%, and somebody from that state is in the top 15%. Given ~10,000 applicants, the top 7% is 700 people...there are only 50 states. It's probably only a handful of people who get bumped up, and it gives all the Senators the chance to support their state (or at least one person from it) by supporting NSF.

Edited December 22, 2010 by BlueRose

[t_ruth]

Big deal or not it does make me wish I were from Alaska or somewhere equally underrepresented

[Eigen]

I submitted an application this year in Biological Chemistry- I think it was reasonably strong, but I'm a bit worried about not having enough previously completed data for the project. Since I'm applying as a second year, I think the competition leans more towards projects that are partially complete, and most of my first year was spent on a related project, with only the last few months spent on the one I'm proposing. That said, it meshes well with our previous research, and I think the letters I got spoke to my ability to get the project done by collaborators in different departments.

Nothing left but to wait and see, now.

[t_ruth]

I've never heard that Eigen. I've always heard it should be a proposal for something that is NOT started...

[Krypton]

Since I'm applying as a second year, I think the competition leans more towards projects that are partially complete

I don't think that's true. At least in biology, all 2nd-years have been working on their projects for only a few months now. I think, as with college senior- or 1st-year-submitted applications, the foundation for what you're proposing just has to be there.

I've always heard it should be a proposal for something that is NOT started...

That's not the case, either. I know people who have applied with proposals for which promising preliminary data (i.e., for Aim 1) has already come out. I did this myself, and one reviewer cited this as good evidence my project had the potential to even move past its initial stage.

Edited December 22, 2010 by Krypton

[Eigen]

All I know is what's filtered down from people who've applied previously, which is quite honestly not much.

That said, the successful applications I've seen (mostly biology) seemed to have very nice preliminary data.

But I completely agree, I think that the foundation for what you're proposing (including it's feasibility with the facilities and collaborations available to you) and its potential are the most important parts.

It does have to get a bit more concrete as you go up in years, though- compared to the hypothetical project with one possible professor in the college senior applications, or with 1st year applications when some applicants still haven't chosen a research group.

[BlueRose]

...I'm a bit worried about not having enough previously completed data for the project. ...most of my first year was spent on a related project, with only the last few months spent on the one I'm proposing.

I think it would be fairly unusual to have more than a few months' worth of data, even as a second year. Everyone has coursework, qualifiers, rotations, etc - from what I understand, the first year of grad school goes by with little/no research progress, even if you're good.

Some of the sample proposals I've seen did have data - at the extreme, I remember one which actually had a data plot. But even then, I don't think there was more than a few months' work represented. What did seem to be important was a sense that the project was "shovel ready" - that you could actually go out and start working on it tomorrow, because you'd planned it so well. Data just means you're so ready that you've already started.

It's an interesting balance. The proposals are supposed to be a test of your ability to design a convincing research project - you don't actually have to do what you say, and the NSF doesn't care if you do or not, so it doesn't matter if your idea is wrong (as long as it's not so wrong that the judges notice). On the other hand, it's easier to be convincing if you have some data to support your claims.

Also, if you did have a non-trivial amount of data, where would you put it? Two pages is nothing. I have only the slightest bit of preliminary data, which I mentioned in a paragraph addressing potential problems - and I had to cut the whole thing, because there was just no space.

On a completely different note, I wonder how they match proposals to reviewers. What little I've heard implies that they grab a folder at random and have a go at it. That seems plausible, but I kind of hope it's not true.

I applied in Computational Biology, which is shoehorned in with the rest of molecular biology. And I have a really computational project - seriously, I've got about half an Aim (out of four) that requires a wet-lab. This makes me nervous. I tried to make it as clear as possible...I imagined writing it for my boss, an excellent cell biologist who doesn't do computers. But there's only so much WTF-mitigation one can do.

[Eigen]

On a completely different note, I wonder how they match proposals to reviewers. What little I've heard implies that they grab a folder at random and have a go at it. That seems plausible, but I kind of hope it's not true.

I applied in Computational Biology, which is shoehorned in with the rest of molecular biology. And I have a really computational project - seriously, I've got about half an Aim (out of four) that requires a wet-lab. This makes me nervous. I tried to make it as clear as possible...I imagined writing it for my boss, an excellent cell biologist who doesn't do computers. But there's only so much WTF-mitigation one can do.

I'm similarly worried- my proposal balances between traditional organic synthesis and more detailed molecular biology- I wouldn't expect either group to understand the other end of the proposal very well. My boss doesn't even really get most of my bio work, I have a second professor at our med school I collaborate with for that. And he really doesn't understand the synthetic and design work I do.

I think we have to hope that if it's well enough laid out that it will be ok- but as you said, with only two pages, there's only so much to explain!

[olive81]

This is going to be another long 3 months. At least is the last time I have to do this for this particular award.

[t_ruth]

This is going to be another long 3 months. At least is the last time I have to do this for this particular award.

the last chance thing is what makes the wait (and the looming likely disappointment) all that much worse!

[tracklt16]

Hey I applied under the social sciences-- political science. Does anyone know how we will find out if we have won the award?

[t_ruth]

They will send out an email once they've made the decisions.

[tracklt16]

thank !

They will send out an email once they've made the decisions.

[t_ruth]

The panelists meet in February, right? Starting to get antsy

[scyrus]

I applied as well, under Computer Science =)

Does anybody know if the NSF requires/encourages/ignores fall semester grades?

[t_ruth]

I don't think they consider fall grades. Transcripts are due before they come out...

[ChemistryGirl]

"It's an interesting balance. The proposals are supposed to be a test of your ability to design a convincing research project - you don't actually have to do what you say, and the NSF doesn't care if you do or not, so it doesn't matter if your idea is wrong (as long as it's not so wrong that the judges notice). On the other hand, it's easier to be convincing if you have some data to support your claims."

What do you mean that it doesn't matter if your idea is wrong? After I submitted my proposal I found some literature showing that I picked an inappropriate substrate to work with...I kind of figured I'd killed my proposal even though the background and idea was good.

[BlueRose]

What do you mean that it doesn't matter if your idea is wrong?

In "real" grants, you have to submit progress reports and such, and failing to meet the goals you stated will hurt your ability to get more funding. For these grants, you do have to tell NSF what you're up to, but they'll rubber-stamp it as long as you're still in basic science. If you're wrong, you do something else. I think most people do anyway - heck, I'm not even sure where I will end up for grad school, much less which lab and what project.

That said, you're still going to have to convince people that your idea is plausible. But for better or worse, the disciplinary spread is pretty broad in these panels - and remember, they're reviewing the whole thing in 15 minutes, max. I'm not sure they'll catch subtle mistakes.

I also said something rather stupid that I wish I could take back. There was a statement of the form "all known A have property B"...all but one, and I found that one. (Yes, there was some very late-night editing involved.) But unless I have the bad luck to find someone who remembers my poster on this better than I do, it probably won't matter. The argument holds - at least it did before I started deleting stuff - and it's a minor point anyhow.

Edited January 21, 2011 by BlueRose