How SHOULD Applications Systems Work?

[TK2]

So here's kind of a broad general issue i've been thinking about while waiting around and procrastinating on my actual work. This applying thing is tedious, exhausting, expensive, time consuming, seemingly arbitrary at times and strongly opaque - this coming from the perspective of applicants. Obviously programs set it up in ways that they feel work for them, which might be frustrating for us, but probably have their (pretty good) reasons - but some reasearch on things like the GRE or genuinely diverse admissions shows mixed results on the university side as well. (Just saw an article about a study that found that only some tiny percentage of MBA programs, for example, met what they considered their own criteria for admissions processes.)

Other systems seem to be more diverse - in the UK there's more formal space for contact with POIs and research proposals as part of the application. In Europe PhD applications are typically more like job offers, stepping into a role in an existing project. In Israel it's highly informal, mostly at the level of developing a strong personal contact with a specific professor who agrees to supervise and/or has funding and a lab, etc, and the applying to the university bit is somewhat more of a formality. I don't know how it works anywhere else.

I'm curious how people think this whole rigamarole can be improved? What are you finding annoying but acceptable? (For example, I get that admissions criteria can be vague. That's just how it is. I believe departments when they say they have no single portrait of an ideal candidate. (Maybe I'm naive)) Compared to what's annoying but seemingly has no good reason? (The GRE mysteries, for example, seem pointless to me - if there are internal cutoffs, they should be advertised. If it's genuinely barely looked at - and studies appear to show there's hardly any correlation between the GRE and, say, likeliness to complete a PhD - stop requiring it. If it's used in as a factor against a poor GPA, say exactly that and mean it.) Contrarily, what works just fine, as far as you're concerned?

[TakeruK]

To add to your list, I would say that for some fields, Canada is similar to the US and in others (mainly physical sciences but probably more that I don't know about), it can range from as informal as how you describe Israel to something more like UK/Europe. 

From the "other side", I would say you are right that there is no single portrait of an ideal candidate. This also means, to me, there is no single right way to do applications! I'm also interested in the admissions process and we've had many conversations on our campus about not just the admissions process but the recruitment process too. This is especially important because we are striving to ensure we do not exclude any qualified students because of their race or gender. So, we examine the way we set up the admissions process and what kind of recruitment we do to ensure that we are a desirable place for everyone. (Note: I'm now at a US school and all of the following is under the US system).

I think more transparency in the process would be helpful. However, I also think it is better to have no information than misleading or incorrect information. For example, I think many applicants often want to know things like: "what is the average GPA of an admitted student?". However, I don't think providing this information (at least for my program) will help. I actually think it will be very misleading. First, we only make 6 or 7 offers per year, so the annual average will have large variations. Second, there are lots of factors that vary from year to year so you can't really do multi-year averages either. Third, we evaluate candidates holistically and look at their total profile, so a candidate might be well below average GPA but still get an offer while another candidate with a perfect GPA may not get an offer due to other factors.

I'm just using GPA as an example here, the same would apply to virtually every admission criteria. In any case, providing this information has more harmful effects than beneficial ones, in my opinion. If we publish averages and they are high, we might lose qualified candidates that are discouraged from applying. On the other hand, if we publish them and they are low (or if we publish the lowest accepted score in each category) then we might have people that would have little chance of acceptance applying and wasting their time and money. 

In addition, it's important to note that there are many factors completely outside of the candidate's control. For example, this year, applications to one particular subfield was very popular. So much that we had to turn away students who would normally get into the program if they had applied in another year or if they had applied to a different subfield. This is because we only have so many people working in this subfield and the profs only have time for so many students. This is something we cannot communicate ahead of time because we don't know how popular each prof/field will be. Also, having these highly qualified candidates be rejected would greatly skew our averages or distributions if we were to publish them. Another example was a few years ago, during the government shutdown times, when people were worried about losing their government grants. We definitely admitted a smaller class than normal that year. Also, we often expect 50% to 60% of our offers to be taken so we aim for a class size of ~4 and we make ~6-7 offers. Some years, we get a very large incoming class! So, fewer offers are made next year. All of these factors mean that some rejected students would actually have been accepted if only it were another year.

Overall, I would estimate that factors that are outside of your control might make up as much as 1/4 of the "weight" of the admission decision. Note: This is just based on gut feeling, not any data. This is common through the rest of academia. When you apply for grants, fellowships, postdoc positions, talk slots at conferences, etc. there will be some amount of randomness present. There are always way more qualified candidates than there are positions/openings/awards available. Knowing this also helps make rejection easier to handle and you'll see that even the superstars in the field have faced rejections and failures.

That said, I do think there is room for improvement in the current system. I think we are currently providing far too little information to candidates about what we're looking for. It's frustrating because this means you either have to "be in the know" (e.g. family member in academia, or have great mentors) or be able to access resources like these forums or other online places to learn what are the norms in academia. 

If I am ever in the position to make these types of policy changes, I would want to make the following more clearly stated: 1) what are the broad themes of experience, expertise and knowledge desired, 2) what is the relative importance for each of these and 3) what are we looking for to evaluate/determine these themes. So, kind of like a rubric, but not quite. One example is the Canadian government grad fellowships, e.g. http://www.nserc-crsng.gc.ca/Students-Etudiants/PG-CS/CGSM-BESCM_eng.asp (scroll down to "Selection criteria"). I think communicating this sort of information is important since some students without experience in academia may not know a lot of what I just wrote above and then not write appropriate SOPs etc. I firmly believe that if we are going to "test" candidates (e.g. via an application process), we should let them know what we are expecting!

Summary: I think the actual applications themselves are working well, but the way we communicate these expectations could use improvement!

[TK2]

3 hours ago, TakeruK said:

Summary: I think the actual applications themselves are working well, but the way we communicate these expectations could use improvement!

Thanks for that thoughtful response!

Yes, I would agree that its more about what's communicated to applicants, and how we understand application, rather than what the actual process any given school or department puts those application material through.

Better information I think would be a huge help, and I wonder why departments tend to be so stingy with it, particularly in official formats. Often the most useful, frank information about what they're looking at both in general, and from a specific piece of the application - like, a minimum of such and such maths classes, or specific things to mention in an SoP - you can find is buried in a PDF from 2007 posted on the dean of graduate students' personal website under a tab titled 'My favorite 70's Latvian polka vinyl records ranked and other personal miscellany, beware the leopard.' (I'm only sort of kidding.)

Its not that it seems to be a secret or something individual faculty are particularly wary of sharing, but no one seems to really care to put up that level of detail the official FAQ. I believe the reasonable applicant can understand that even if that section will list Linear Algebra or whatever now, that isn't a hard requisite, but can take into account that its something committees like to see, or sub-specializations that they're applying to like to see, etc, and will weigh their options accordingly when deciding where to apply. (or sign up for some summer classes to improve their chances based on actual knowledge rather than guess work, etc.)

Speaking of Canada, I really appreciated U of Toronto's Geography Department's approach, for example - http://geography.utoronto.ca/graduate-geography/application-admissions/supervision/. A tab titled very clearly 'Finding a Supervisor' and listing, in a single place, in well ordered paragraphs, what all faculty who are interested in taking on students are working on and who they're looking for. It's the only page of its kind in existence that I am aware of, and should be treasured as a glowing jewel. Mind, I don't know how accurate or for real it is. Maybe they never update it. Maybe its a wishlist. Who knows. But it gives a better sense of what the hell 'departmental fit' is than anything else I've seen.

On that note, I think the process of contacting POI's before application needs to be either officially formalized or utterly deformalized, though I don't know that erasing the value of personal contact is ever truly possible in any field. But it would help if departments at least had genuinely open guidelines about what is acceptable, what the protocol is, etc. (The way UK universities do.) Even on the level of individual professors - stick it up on their profiles on something. This person prefers not to be contacted in advance but only to read your application/this person finds it acceptable to be contacted, please attach your CV and write no more than 100 words about what you intend to pursue in you PhD when you email them (...and please get in touch with such-and-such administrative person who is not maybe doing field work on a sheep farm in Kyrgyzstan all summer if you have not heard back after three emails or three months...) etc.

For example, I recieved an offer to be deferred to next years application from one university in the US where I messed up getting some stuff in by the deadline, and the very nice admin person sent me an email saying I now have 11 months to improve my application and 'build relationships with the faculty'. Wait what? Was I supposed to be building up relationships with the faculty? Is this a very important factor in my application? It does not SAY anywhere that this is a program that requires contact with faculty before hand. What are the channels via which it is acceptable to build up relationships? Etc. So anyway, put that out there if this is actually a huge criteria in how you select your students. Its never going to be perfect, but make it more straightforward/encouraging, because otherwise there's this huge advantage to people who are, well, plucky, and possibly who have the close inner advice that really, this is how it works.  The UK system requires just as much finesse and chasing people down and sending nail-bitingly worded emails, which i'm fine with because its a skill we all have to learn sometime, but it says pretty clearly on most applications guidelines that you have to. With the US system yo have no idea if emailing a POI is actually helping your cause, or basically essential, or shooting yourself in the foot, because maybe they hate that or its just irrelevant and you're wasting time and energy.

[TakeruK]

Yeah, I agree with you on being less stingy with information. I know that at some places, there is the same problem when it comes to quals or other exams: where's the official info for current students to know what is expected?! Weirdly enough, it's not meant to be a secret because profs are happy to talk to you about it and they get surprised and say things like "Isn't this on a website somewhere?"  I know this is a slightly different problem since as current students, we can walk into our profs' offices while applicants have no such access. But it's the same idea that website information is not updated or people think it's clear when it's not!

In addition, I would not really trust websites that have information that isn't timeless, like who's taking students, what classes are being offered and what are each profs' research interests. I know that at one school (which shall remain nameless to protect the guilty), this info is horribly out of date despite students requests to update it. Classes that have not been taught for 5 years were on the list and profs that have moved on to different topics had very old stuff listed.

One part of the problem is the lack of efficient communication between the faculty and the people who update the website. Did you know that the faculty rarely update this information at many places? My spouse works as an admin person at a University and one of her jobs is to update the website. The faculty profiles get updated once per year and she has to really bother the profs in order to get them to give her updated info. She is persistent and I think 90% of the people are updated every year (but of course a prof might just resubmit last year's materials or other outdated materials). I know at other places, the admin staff may not be as persistent (or the profs more stubborn!) so the profs may not be updated. 

The University of Toronto example you have is a good one. It seems to have some model where the admissions committee solicits this info from profs who want to recruit students. These profs are especially motivated to get new students so that seems to be a good way to actually get info from them. (As you said though, it could be outdated resubmissions and you'll never know). One potential problem with this approach, at least in my current department, is that profs are generally pretty open to working on whatever is interesting if a good student wants to take a different direction. Students in my program tend to start taking charge of the projects and lead them in their own way after 2-3 years here, so it's not always true that a prof has a list of interests or projects that they want to find students for. Instead, they sometimes just want to find good students and then determine what work can be done afterwards.

For the feedback you got from the admin person, it might just be a generic response that doesn't mean anything. I don't think you should read as much into it as you seem to be doing here.

Finally, I can't speak for everyone but I think you don't have to worry about negative effects of emailing or not emailing your POI in the US. I emailed all of them in my applications because that's the norm in Canada. 1/3 of them replied positively and we had a good short chat about research and applying. 1/3 gave me a canned response like "Thanks for your interest in our program and my group. At Department X, we don't admit students into specific groups but instead only admit them generally to the department. I would love to talk with you more about potential projects after you receive admission to our program. Good luck!" and the last 1/3 did not respond. That's okay though, I was generally able to get what I needed from each school. I don't think have standardized methods like "email" or "don't email" because people aren't going to agree on what they want. But I don't think emailing when they secretly did not want email will hurt you since if they are that busy, they aren't going to spend their time remembering who you are and then penalizing you later on. They will just ignore it and forget you a few minutes later (sorry! but probably true). They might pull it up again if they see your name in a shortlist or something and it rings a bell. So it's still good to always be professional etc. in these correspondence.

[TK2]

Haha yeah, that last post turned into venting by the end of it.

I disagree on the POI* contact thing. Not how departments or individuals choose to use/ignore/whatever, but how these things are communicated to applicants. I think its an area where expectations are needlessly fuzzy, frustrating and with little information provided about norms, expectations, role in the process etc. This is exactly the kind of thing that isn't very important (in the US system, at least) while being unclearly positioned as a requirement nontheless. In other words - stick it in the FAQ if its a thing.

Personally, it would never have occured to me to contact anyone in the US if I hadn't had people with the relevant knowledge to approach who egged me into it. It was pretty random, just the professor I had a good relationship with happening to favour the lots-of-contact approach, I think. The role of this kind of stumble-upon advice should be minimized, not expanded.  My experience in working in a number of countries at very diverse levels of development is that a huge gap in people's academic ambitions - people who on paper have equivalent BA or even MA degrees - is in the soft skills. Like what's expected when contacting a professor. Lacking public guidelines, that gap is magnified even more.

I wonder how comfortable anywhere would be actually publishing their deliberations, with indetifying information redacted? Not at all, I imagine, but there's an article out there somewhere of a study of the role of diversity in admissions, really, and the author sat in (and quotes) during admissions committees meetings. It's vaguely horrifying/titillating reading, at this point. It felt like that time I accidentally overheard part of a teacher's staff meeting about other kids when I was in middle school...equal parts fascination and transgression.

*What does that stand for? I've never figured it out. Professor of Interest?

[TakeruK]

12 hours ago, TK2 said:

I wonder how comfortable anywhere would be actually publishing their deliberations, with indetifying information redacted? Not at all, I imagine, but there's an article out there somewhere of a study of the role of diversity in admissions, really, and the author sat in (and quotes) during admissions committees meetings. It's vaguely horrifying/titillating reading, at this point. It felt like that time I accidentally overheard part of a teacher's staff meeting about other kids when I was in middle school...equal parts fascination and transgression.

I know that most people would not want this at all. It's also why we waive our rights to use FERPA to view our student records and thus access our letters of recommendation. It's to allow our professors to write openly about us and for the committees to know that our letters were written in this way. I think if deliberations were recorded, it would severely limit open and free discussion. 

I also struggle with the balance between "secret" information and the effects it has on diversity etc. I agree it is problematic and it should be addressed. But I don't think making deliberations public is the way to do it. Instead, what I'd imagine would happen is that there would be a lot less discussion and everyone would just make their decisions privately and there would be even less transparency. You can't even have other committee members noticing bias (intentional or not) and calling out their colleagues for it. 

I think there are better ways to address this, such as more transparent and descriptive criteria. I wrote about this above mostly for the benefit of the applicant, but I think it's important for uniformity among the reviewers too! With an explicit list of evaluation criteria, (un)intentionally biased reviewers would have to justify their evaluations with reference to the uniform criteria. Studies have shown that this does reduce bias in decisions, especially unconscious bias. Even training in the effects of unconscious bias does a lot.  

Regarding the FAQ on contacting profs, sure I think it's a good idea for most departments to have a FAQ for the department in general. I thought you meant one specifically for each professor. Most professors will not prioritize the time required to keep a recruitment page up to date and active. I've seen this question in various departments before, and the FAQ answer is usually helpful to know the general mood of the department, but it often comes with a qualifier that some profs might expect different things. 

I understand the frustration of not knowing the "unwritten rules". I think that teaching these "rules" to everyone is the way we can achieve fairness and more opportunities for everyone. I know that some physics profs have started teaching this to their 2nd and 3rd year undergraduates as part of a "seminar" class or "research methods" class. I think it makes more sense for schools to teach this to their undergraduates than for applications to have every single piece of information possible. This type of class would have been really helpful for me as an undergrad. I had no idea what grad school even was and I was lucky my undergrad advisor was a great mentor or I would have missed out! If I ever get into a faculty position, I hope to be able to push the department to allow me to teach a 1 hour per week, "research skills" type class where we learn things like these "unwritten rules", hear about different things you can do with your major (academic and non-academic), learn how to do a literature review etc.