Machine Learning - Statistics vs CS PhD

[harry_stats]

I am going to be applying to PhD programs for next year. My background is that I went to an Ivy League school and majored in OR. After college, I am working as a software engineer and have taken some math classes to beef up my profile. I am planning on taking the Math GRE next fall.

My interests lie in machine learning and modern statistics. Some areas that I may want to work on down the line are optimization, variational inference, robustness and theory of deep learning. The research I find most meaningful both develops new algorithms and provides theoretical guarantees. I am not interested in working on specific applied areas such as NLP, computer vision, robotics.

Should I be applying to statistics or CS programs? What are the best programs to do more statistically rigorous, theoretical machine learning research?

Looking at course requirements in statistics programs, it looks like many schools (Stanford, University of Washington, UChicago, etc.) require advanced probability or measure theory sequences which may prove useful but are much more important for someone doing research in classical statistics, program evaluation or stochastic models. Course requirements in the CS programs at these same schools look incredibly flexible, allowing me to take whatever courses I would want across several departments. But I am finding that most of the research papers I find interesting tend to be by professors with joint CS and statistics appointments and many times require a very deep statistical background.

Edited April 30, 2020 by harry_stats

[StatsG0d]

I suggest that you read this thread, particularly the posts by myself and @Stat Postdoc Soon Faculty.

If you have further questions, I'm happy to weigh in.

[Stat Assistant Professor]

OP: Assuming you did well enough in your undergrad and are performing well in your current math classes as a non-degree seeking student, you can probably get into a pretty decent Statistics PhD program, even without the Math subject GRE. Tbh, the math subject GRE isn't very useful or indicative of anything in most cases (it tests on subjects like abstract algebra and topology that are pretty much useless for Statistics). I think only Stanford requires it, and I think they mainly use it as a screening tool since they get so many outstanding applicants. 

On the other hand, if you want to get into a CS program, you're definitely going to need research experience and preferably at least one second-author conference paper. To be competitive enough for PhD admissions in CS, you would probably need to either work as an RA in a lab for a year, or get a Masters in CS where you do some research on the side for a professor, in addition to your coursework.  You could bypass the Masters and research experience completely by mainly applying to Statistics PhD programs (especially ones where there are faculty who hold joint appointments in both Statistics and CS). Given your interests, you could do very well in a Statistics department that has faculty working in those areas. Columbia (Dave Blei), UC Berkeley (Michael Jordan), and University of Michigan (Jeff Regier) come immediately to mind.

Edited April 30, 2020 by Stat Postdoc Soon Faculty

[bayessays]

Agreed with above. You'll really have to dig for people like the above working on deep learning in stats departments. UT Austin is another department that has people working in the areas you listed. 

[harry_stats]

Thanks! Two follow up questions:

1. In the older thread, @StatsG0d mentioned that if my interests are just in ML it may make sense to find stats departments that have ML embedded in the curriculum. Any sense of what are the top departments where that is the case? 

2. @Stat Postdoc Soon Faculty mentions that you need a masters or research experience to get into a CS program. I really do not have either of these, so not sure that route would work out for me. Any reason why there is such a discrepancy in CS and stats programs admission criteria? My thinking is that a quantitatively strong student (with background in probability, analysis, discrete math, algorithms, etc.) would be able to make meaningful research contributions in either field. Since the ML research in CS depts is actually more applied, it seems like proving research potential there should be easier? It seems that it would be more important to have a masters (proving mathematical maturity or research potential) in statistics given the research is a bit more theoretical. 

Edited April 30, 2020 by harry_stats

[bayessays]

I think a lot of it has to do with the nature of research in the fields.  CS departments are centered around conferences, and people start doing research early (you're generally admitted into a particular lab), so they want to make sure you can do that. Statistics departments fund you at the department level and you aren't expected to do research early because there is a lot of coursework to teach you the basics of the field.  Almost nobody does real statistics research as an undergraduate, so there can't be an expectation of prior experience. 

[icantdoalgebra]

Actually the criteria for getting into ML are changing somewhat, or at least for Berkeley. Part of the reason why the publishing requirement is so high to get into PhDs is that the bar for publishing in a top conference like ICML or NeurIPs is far lower than publishing a paper in a top stats journal like AoS, JASA, or JRSS-B. ML has this reputation of being a field where you just have really large labs and just churn out as many papers as you can every year (Levine at Berkeley who does RL research submitted 40+ papers to a conference on RL and had I think 24 of them accepted). This creates a issue with the people who review the papers, as there simply is too many to review each one thoroughly. Berkeley to an extent has recognized this problem (one professor even told me that conferences in ML are filled with "garbage" papers) so they now value letter of recommendations far more than publishing record. But again, usually the best letters of recommendations comes from Professors who you've done research with. 

As for Stats vs ML, I think you should apply to both programs where you can. For example, CMU lets you apply to multiple programs so there's really no reason not to do so (unless the financial requirement creates undue hardship). Even if you have to choose, a lot of schools have faculty in multiple departments because they recognize that the work is very similar and you can work with these faculty members regardless of department. Although it also depends on how strongly you want to do ML, as you would be hard pressed to find statistics faculty that aren't joint CS professors who do deep learning; everything else you would be able to find in a statistics department.

[StatsG0d]

16 hours ago, harry_stats said:

Thanks! Two follow up questions:

1. In the older thread, @StatsG0d mentioned that if my interests are just in ML it may make sense to find stats departments that have ML embedded in the curriculum. Any sense of what are the top departments where that is the case? 

Obviously, CMU's statistics / ML program would be a great fit. Other than that, I'm not sure there are many statistics programs that offer ML as foundational courses. Perhaps the best thing to do would be to target departments that have strong ML faculty and whose students take the qualifying exam after the first year. That way, you'll receive the foundational statistics knowledge you'll need without having to go super in depth on topics that won't relate to ML, and can get started taking ML electives earlier.

Another thought is to look at the biostatistics departments that are doing some good research in ML / precision medicine. Washington (Shojaie), NCSU (Davidian, Laber), UNC (Kosorok, Zeng), McGill (Moodie) are some of the "heavy hitters" that come to mind that publish in top journals.

[harry_stats]

@StatsG0d - thanks! I did some extra research on programs that would allow me to take fundamental statistics courses early and get involved in ML research after year one. From what I gathered, the best choices as follows,

Stanford statistics (Tibshirani, Hastie, Duchi, Ma)

Duke (Dunson, Rudin, Parr)

CMU statistics / ML (entire list of statml theory group faculty)

Berkeley statistics (Yu, Wainwright, Jordan, Steinhardt)

Univ of Washington statistics / biostatistics (Shojaie, Witten, Harchaoui, Kakade) - assuming you place out of the masters level coursework

Univ of Michigan statistics (Nguyen, Regier, Tewari)

Are there are any other top 5-10 programs I am missing here? The other programs in the rankings including UPenn, Harvard, Chicago do not seem to fit the bill in terms of providing the expertise or ability to do meaningful ML research.

[ZNtheory]

3 minutes ago, harry_stats said:

@StatsG0d - thanks! I did some extra research on programs that would allow me to take fundamental statistics courses early and get involved in ML research after year one. From what I gathered, the best choices as follows,

Stanford statistics (Tibshirani, Hastie, Duchi, Ma)

Duke (Dunson, Rudin, Parr)

CMU statistics / ML (entire list of statml theory group faculty)

Berkeley statistics (Yu, Wainwright, Jordan, Steinhardt)

Univ of Washington statistics / biostatistics (Shojaie, Witten, Harchaoui, Kakade) - assuming you place out of the masters level coursework

Univ of Michigan statistics (Nguyen, Regier, Tewari)

Are there are any other top 5-10 programs I am missing here? The other programs in the rankings including UPenn, Harvard, Chicago do not seem to fit the bill in terms of providing the expertise or ability to do meaningful ML research.

I think Prof. Weijie Su who is an AP in Wharton in UPenn has started to do research in the area of fundamental of deep learning and published some papers on conference like Neurips. I think you should check him in case you are interested in his work.