Determining Rankings

[edvolkov]

Regarding faculty from top-10 schools, this is relevant:

http://pages.cs.wisc...mni_matrix.html

But this statistics is too old I guess - there are several mistakes.

Edited May 16, 2011 by edvolkov

[symbolic]

But this statistics is too old I guess - there are several mistakes.

It's from 2005. I seriously doubt that much has changed, or that the few faculty hired since then drastically buck the trends shown here. While there is room for error in that data, and while the data has probably changed a bit since it was collected, it's clear that there is a trend.

[edvolkov]

It's from 2005. I seriously doubt that much has changed, or that the few faculty hired since then drastically buck the trends shown here. While there is room for error in that data, and while the data has probably changed a bit since it was collected, it's clear that there is a trend.

OK, I don't put in doubt that there is a trend. I have the same point of view on academic placement. While there is a very huge gap between several (5-7) top schools and others, the difference between other schools (up to top-50 I guess) is very slight.

Btw - there is statistics for alumni since 1995. http://pages.cs.wisc.edu/~estan/alumnistatistics/top25/AlumniAA_matrix.html

[symbolic]

Yeah, I'd say that's definitely true. Beyond a certain threshold, the differences in whatever measure tend to plateau pretty quickly. Or it might be just that we tend to gloss over what we don't see as "elite" and then (unwittingly) flatten true distinctions. But I like to think I'm not that snobby.

[jetru]

What about for people who want to get into industry after the MS? How much does ranking matter there?

[Bleu]

There are a few worldwide rankings of universities out there, such as

Shanghai Jiao Tong University's Academic Ranking of World Universities

Times Higher Education rankings (they don't have a CS specific ranking)

Quacqarelli Symonds' rankings (they don't have a CS specific ranking)

Those are three that I've come across, I'm not sure if there are others. I can't speak to the value of these rankings and some of their rankings look a bit suspect, so use it FWIW.

I have seen another ranking which relies merely on research publications. It claims to be more reliable than the above rankings. It also has a CS specific ranking among other field and subject specific ones. Another interesting thing about it is that unlike other rankings, it covers 300 universities in its subject specific rankings. Here's the link:

http://ranking.heeact.edu.tw/en-us/2010/homepage

[symbolic]

I have seen another ranking which relies merely on research publications. It claims to be more reliable than the above rankings. It also has a CS specific ranking among other field and subject specific ones. Another interesting thing about it is that unlike other rankings, it covers 300 universities in its subject specific rankings. Here's the link:

http://ranking.heeact.edu.tw/en-us/2010/homepage

These rankings are really iffy: look at how many Asian universities dominate in the top 10 or top 15. I'm not saying that's impossible, but the rankings came from an Asian organization, and no other ranking (even the ARWU) seems to put that many Asian universities so high. Not just in the CS ranking, but most of their subject rankings as well. I know that Tsinghua, etc. are really great schools, but come on, they definitely don't dominate like these rankings purport.

[BKMD]

It's really hard to come up with quantitative ways to rank schools. Publication counts might be meaningful, but you also want to consider how often the papers are cited, where they're being published (high-acceptance regional conferences vs competitive high-impact venues), etc. And you want to somehow normalize by the size of the department, since obviously larger departments will have more publications. It's hard to measure this, and you have to make arbitrary choices. And then when you try to combine several factors (e.g. publications, admissions percentages, etc.), you have to come up with a way to weigh the different factors. At the end, you're going to have a ranking that's highly sensitive to arbitrarily chosen parameters, and it's almost meaningless (though you can still get a sense of different "tiers" of schools).

The US News rankings are nice because for grad programs, they're based only on opinion surveys. No parameters to tune; just a straightforward way to directly measure the perceived prestige of a school. If you're interested in the reputation of a department, this is what you want, and these are the rankings people are usually referring to.

When choosing between schools, I think the thing to look at is the actual job placements of alumni of the particular labs (not departments) that you plan to work in. Assuming your end goal is a particular career, then I don't see why anything else should matter. Obviously top schools are going to have better job placements on average, but you can find plenty of labs at lower-ranked schools with stellar job records (often at schools that you wouldn't guess unless you were familiar with that particular research area) and conversely, you can certainly find research groups at top schools that have a mediocre record. You'll also find that some schools tend to be oriented more toward industry vs academia which might be something to be aware of if you have a sense of which direction you want to go.

I think the advice to go to the best school you can get into is "on average" a good heuristic to follow, but it's not always black and white and it might be the wrong thing to do in some circumstances. If you dig just a little bit deeper, you should be able to figure out how good of an education you'll really get and what kind of career placement you can expect.

[frenzydude]

It's really hard to come up with quantitative ways to rank schools. Publication counts might be meaningful, but you also want to consider how often the papers are cited, where they're being published (high-acceptance regional conferences vs competitive high-impact venues), etc. And you want to somehow normalize by the size of the department, since obviously larger departments will have more publications. It's hard to measure this, and you have to make arbitrary choices. And then when you try to combine several factors (e.g. publications, admissions percentages, etc.), you have to come up with a way to weigh the different factors. At the end, you're going to have a ranking that's highly sensitive to arbitrarily chosen parameters, and it's almost meaningless (though you can still get a sense of different "tiers" of schools).

The US News rankings are nice because for grad programs, they're based only on opinion surveys. No parameters to tune; just a straightforward way to directly measure the perceived prestige of a school. If you're interested in the reputation of a department, this is what you want, and these are the rankings people are usually referring to.

When choosing between schools, I think the thing to look at is the actual job placements of alumni of the particular labs (not departments) that you plan to work in. Assuming your end goal is a particular career, then I don't see why anything else should matter. Obviously top schools are going to have better job placements on average, but you can find plenty of labs at lower-ranked schools with stellar job records (often at schools that you wouldn't guess unless you were familiar with that particular research area) and conversely, you can certainly find research groups at top schools that have a mediocre record. You'll also find that some schools tend to be oriented more toward industry vs academia which might be something to be aware of if you have a sense of which direction you want to go.

I think the advice to go to the best school you can get into is "on average" a good heuristic to follow, but it's not always black and white and it might be the wrong thing to do in some circumstances. If you dig just a little bit deeper, you should be able to figure out how good of an education you'll really get and what kind of career placement you can expect.

That is an amazing answer. +N !

[Bleu]

These rankings are really iffy: look at how many Asian universities dominate in the top 10 or top 15. I'm not saying that's impossible, but the rankings came from an Asian organization, and no other ranking (even the ARWU) seems to put that many Asian universities so high. Not just in the CS ranking, but most of their subject rankings as well. I know that Tsinghua, etc. are really great schools, but come on, they definitely don't dominate like these rankings purport.

I think that one of the facts that puts some Asian universities top in this ranking is not considering the size of the department. It's obvious that a university with a lot of profs and PhD's and post docs generates a lot of publication, which can cover their even low quality in the final ranking score. But by considering criteria like publishing in Science/Nature or winning Turing award, a ranking like ARWU eliminates these less known schools from being ranked among top schools. This criterion has its own drawbacks too, for example a university having a prof who has won a prize some while ago would rank higher than a much better school which lacks such prize winner.

[barber5]

I have been looking at the breakdowns here which allow you to rank authors, publications, conferences, journals, and institutions in 24 sub-areas of CS based on citations and at least in my areas of study it accords fairly well with the intuitions I have about how things should be ranked.

[symbolic]

I think that one of the facts that puts some Asian universities top in this ranking is not considering the size of the department. It's obvious that a university with a lot of profs and PhD's and post docs generates a lot of publication, which can cover their even low quality in the final ranking score. But by considering criteria like publishing in Science/Nature or winning Turing award, a ranking like ARWU eliminates these less known schools from being ranked among top schools. This criterion has its own drawbacks too, for example a university having a prof who has won a prize some while ago would rank higher than a much better school which lacks such prize winner.

But it's not just ARWU. Virtually every international ranking, whether it includes those same metrics or not, does not place 8 Asian universities in the top 15-20 or so (as is the case in some of the rankings). It quickly becomes obvious that these rankings are heavily biased toward the region's universities. There's probably a good reason that there's not much talk/discussion about these rankings on the internet; they just don't seem credible.