Mathematics for Bayesian Statistics Research

[trynagetby]

Hi all, so I'm very likely to go to Duke Statistics for PhD. My job isn't very demanding and my brain has been slowly rotting away, so I really need to get back into the math groove. Aside from a review of analysis, measure theory, linear algebra, probability etc,  does anyone have any recommendations for what topics are important to study for modern Bayesian research e.g (functional analysis, topology, stochastic processes, graph theory ...). Other from passing first year classes/quals  I'd like to get a general idea of the important mathematical tools out there even I don't have the time to master them.

Edited March 8, 2021 by trynagetby

[Stat Assistant Professor]

So I do mostly research in the area of Bayesian statistics (though not exclusively), and I have done both applied and theoretical research in this area. 

I would say for theory: it is pretty important to know analysis and linear algebra well and to be comfortable with probability theory and stochastic processes. Unless you are doing very hardcore theoretical research (and there are some people who do that), you don't need to know measure theory that well, but you should be comfortable with it. Plus, measure theory/probability theory can be pretty useful for Bayesian nonparametrics. In Bayesian nonparametrics, you frequently replace finite-dimensional prior distributions with stochastic processes (e.g. Dirichlet process, Gaussian process, etc.), and it can be useful to know a little bit of measure theory and probability theory. 

For methodology/applications: obviously be familiar with the Bayesian paradigm, as well as MCMC (Gibbs sampling and Metropolis-Hastings) and maybe variational inference. Most of the time, the posteriors are intractable, so you do need to do approximate inference. It would be useful to be familiar with some of the "classical" models for linear regression, classification, and semiparametric/nonparametric methods for regression/classification. I think once you specialize in a particular research area (e.g. spatial statistics, functional analysis, topological data analysis, etc.), you can learn that stuff on your own. There's no need to study it prior to starting your research, unless you are very interested in it.  

For programming: Be proficient with programming in R and comfortable with using C/C++. Since R can be a bit slow and have a lot of overhead (compared to C/C++), you may prefer to code in C/C++ and integrate this code with R. R is great for creating plots and visualizations, etc., but if you are going to run MCMC (for example), you may prefer to use C/C++ and then integrate this with R, because your code will run a lot faster.

[trynagetby]

Thank you so much! Do you have any textbooks you would recommend for studying Stochastic processes? Is a super in-depth understanding of stochastic processes in terms of martingales necessary (I really struggled with those as an undergrad)?

[bayessays]

It's not a measure-theoretic treatment (such as a class in stochastic processes you might take after a first semester course out of a book like Billingsley), but Resnick's Adventures In Stochastic Processes is one of the most entertaining math books I have ever read - highly recommended.  @Stat Assistant Professor is much more of an expert, so defer to his opinion, but I don't think a huge knowledge of martingales will be needed; the only time I see martingales come up is in some people's theoretical work on Bayesian asymptotics.  Keep in mind that you won't really have to learn much of this stuff that you don't want to -- for instance, if you don't like martingales or functional analysis, most people at Duke will be doing research that is much more applied.  My personal advice is always to just focus on mastering the materials in your classes and if you are starting research, to learn that material.  Unless you really love doing the extra math, the effort:reward ratio is probably not worth it.

[Stat Assistant Professor]

Before starting the PhD, I did review Calculus, probability and statistics (at the Casella/Berger level), real analysis, and linear algebra (including proofs). I think that this was helpful for the classes that I had to take in my first year, since it was fresh in my mind right before starting classes.  

I didn't study any measure theory, stochastic processes, functional analysis, etc. before starting the PhD. I don't think this is really necessary, but if you are very interested in it, it could potentially be useful... though I should note that by the time you start research, chances are you will forget most of this stuff. At that point, you can just relearn what you need for your research and fill in any gaps as you go.  But I do recommend reviewing some of the topics in my first paragraph because you'll be able to use that stuff right away when you start taking classes (rather than possibly needing it one or two years later when you start your dissertation research).

[cyclooxygenase]

Speaking specifically about Duke:

Stochastic processes (specifically applied to MCMC and HMMs and such) get covered as a big part of a second-semester class ("Probability and Statistical Models").  That said, how important stochastic processes are in that class varies by Prof -- if Mike West teaches it then you'll probably eat autoregressive models and stochastic processes for breakfast (and he's a wonderful teacher, so you'll enjoy it!), but if a different prof teaches that class then you might get a bit less stochastic processes and s bit more E-M alg, finite state space MC, and so on.  My point being, what you need to know (at least for the qualifying exam) will be taught, they're not just going to assume you've learned stochastic processes ahead of time.

Martingales are covered in the measure theory course most of the time, although since they're at the end they're liable to get abbreviated if stuff runs over.  Again, you'll learn what you need, at least for quals.  Real analysis is definitely helpful in prepping for the measure theory course, though.  I will note that it might be possible for you to skip measure theory and the intro Bayesian course (mentioned below) if you have strong background there already, although I think that's less likely (/impossible?) to be the case in the future than it has been in the past.  

Duke has you take a Bayesian class first semester (this is the undergrad/master version, but an OK idea of the topics anyways), but even so it's nice to have basic background in conjugate prior sorts of models and coding basic Gibbs samplers by hand (you know what I mean, no JAGS or Stan).  If you have that you're likely fine, although the intro Bayesian class is switching from combined MS/PhD to having a PhD-specific class that may cover more advanced topics going forward.

I'd definitely recommend going over Casella & Berger if you're rusty, since the inference class (which is second semester) is probably the most important class for quals and is taught from TPE2, so there's some assumption you're familiar with C&B and maybe CMT and Slutsky's theorem that aren't covered all that much in the inference class (or they come up right at the end) but that will be relevant on the quals + potentially going forward.  

Code-wise, it's mostly R in classes, although some profs prefer MATLAB.

For topics to learn for research, Stat Asst. Prof's your person, as you already realized.

[Egnargal]

These are all great replies of course and following the advice given in them will certainly benefit you. For me, reading Resnick's A Probability Path was helpful, but now after doing two semesters of probability, I prefer Durrett's or Billingsley's treatment of the material.

 I would also mention that once the program begins, you will likely be consumed by all the coursework, and I found it rewarding to look at some topics that I was interested in but that wouldn't be immediately relevant, knowing that I probably wouldn't have as much time to devote to them once the program began. For me, I took the time to learn more about combinatorics and topology, in particular. (Of course, such topics could become relevant in the context of research, but I investigated them more for my own enjoyment at that point than out of any desire to get ahead of the material.) 

As for Bayesian statistics itself, I am partial to Robert's The Bayesian Choice, which is written in an engaging style and covers a fair amount of ground in establishing the Bayesian paradigm.

[bob loblaw]

 @Stat Assistant Professor I went to a pure frequentist school and have almost no background in Bayes.  
Assuming I have a basic undergrad stats background + upper division linear algebra + real analysis....
Do you have recommendations on Bayesian Textbooks?  I was looking to review the first chapters in  Gelman's "Bayesian Data Analysis" (supposedly the Bayesian Bible). 

Also @Egnargal do you mind comparing Robert's "The Bayesian Choice" to Gelman?

[bayessays]

1 minute ago, bob loblaw said:

do you mind comparing Robert's "The Bayesian Choice" to Gelman?

Gelman is a data analysis book you might use in an applied Bayesian class. The Bayesian choice is more of an intro to Bayesian theory at a higher level and less focused on the applied aspects.

[Egnargal]

Yes, exactly. I am not sure whether I agree fully, but The Bayesian Choice was described to me as a book that reads like a captivating adventure novel and that makes for good, relaxing nightly reading. BDA3 is very good, and now that the chapters and related code are available online, it is even more useful. The first Bayesian book I read was Hoff's A First Course in Bayesian Statistical Methods, which I would still recommend to someone new to Bayesian statistics. There are also nice works, like Robert and Casella's Monte Carlo Statistical Methods or West and Harrison's Bayesian Forecasting and Dynamic Models, which cover more specific areas.

Edited March 9, 2021 by Egnargal