Elements of Statistical Learning (Hastie, Tibshirani, Friedman)

[biostat_student]

I'm planning on self-studying Elements of Statistical Learning (by Hastie, Tibshirani, and Friedman) over the summer. It comes highly recommended by a couple students in my program and current lab (which has a major machine learning component). I'm currently a Biostats PhD student who has finished their first year of graduate coursework (Probability and Statistical Inference I/II, Statistical Methods, Intermediate Linear Models). My goal is to read most of ESL, and if that turns out to be too aggressive of a goal, then parts of it, in order to understand the theory behind a variety of machine learning methods. I don't intend on understanding every word, but I want it to be something useful to me as I move forward into research, and something that I can keep referring back to when needed.

I've read from many sources online that ESL is an incredible textbook, but some sections are difficult to understand even with a decent math/stats background. I'd love to hear any advice that people who have read the book (or attempted) may have. I don't completely know what I'm in for but want to get the most out of it, because it will be a huge benefit to me throughout my PhD program.

I'm especially interested in the following chapters: Ch 3 (Linear Methods for Regression), Ch 4 (Linear Methods for Classification), Ch 7 (Model Assessment), Ch 8 (Model Inference and Averaging), Ch 9 (Additive Models, Trees, and Related Methods), possibly Ch 11 (Neural Nets), Ch 12 (Support Vector Machines), Ch 14 (Unsupervised Learning), and especially Ch 18 (High Dimensional Problems).

I'm open to hearing any thoughts or recommendations, whether general or specific. Here are some questions I had:

• How readable is this book for a someone who has competed the first year of graduate statistics courses (plus math undergrad)?
• What background topics/subjects were particularly useful to have brushed up on before starting to read?
• Were there any supplements/resources you found useful?
• What are some methods for effective reading of the book? (balance between reading/doing problems, setting goals, etc.)
• How much does each chapter rely on the last? Can you skip around or do you need to go sequentially?
• Understanding that the following are very subjective:
  • What sections did you find valuable?
  • What sections did you gloss over? Were any sections more difficult to read than others?
  • Chapters that I haven't listed above that are important? Chapters I've listed above that you didn't find as important?
• General tips for getting the most out of the book or advice for someone before they start reading.

Edited April 20, 2019 by biostat_student

[Stat Assistant Professor]

"The Elements of Statistical Learning" is a good book and I found it to be pretty accessible. If you have already finished the first year of your PhD program, you probably can just start reading it (and then refer to other references when you need to). I think it's a good idea to read this book from start to finish at a high-level (i.e. read it enough to understand conceptually what's going on rather than focusing on the nitty gritty technical details). Then you can refer back to parts of it later and read those sections much more carefully if they are related to your research. I never did any of the problems at the end of the chapter, but I don't think that would have helped much for research anyway.

The best way to "start out" research is not to dive deeply into theory and technical details, but to get a broad sense of the field and then narrow the scope of your project (your PhD advisor should guide you to picking a suitable project that is not too ambitious and not too incremental but "just right" and doable enough to result in a paper or two). Reading through this book, watching tutorial videos, and reading review articles, lecture slides, tutorials, etc. are all great ways to learn the basics/high-level background of statistical learning. Once you start research, your advisor will probably give you a few important/"seminal" papers to read as well, and then it will make sense to delve into technical details, theory, etc.

Edited April 20, 2019 by Stat PhD Now Postdoc

[Bayequentist]

2 hours ago, Stat PhD Now Postdoc said:

"The Elements of Statistical Learning" is a good book and I found it to be pretty accessible. If you have already finished the first year of your PhD program, you probably can just start reading it (and then refer to other references when you need to). I think it's a good idea to read this book from start to finish at a high-level (i.e. read it enough to understand conceptually what's going on rather than focusing on the nitty gritty technical details). Then you can refer back to parts of it later and read those sections much more carefully if they are related to your research. I never did any of the problems at the end of the chapter, but I don't think that would have helped much for research anyway.

The best way to "start out" research is not to dive deeply into theory and technical details, but to get a broad sense of the field and then narrow the scope of your project (your PhD advisor should guide you to picking a suitable project that is not too ambitious and not too incremental but "just right" and doable enough to result in a paper or two). Reading through this book, watching tutorial videos, and reading review articles, lecture slides, tutorials, etc. are all great ways to learn the basics/high-level background of statistical learning. Once you start research, your advisor will probably give you a few important/"seminal" papers to read as well, and then it will make sense to delve into technical details, theory, etc.

Speaking of statistical learning, what do people in the stats community think of deep learning methods such as CNN, RNN, GAN, Deep Graphical Models? Thanks!

[Stat Assistant Professor]

22 hours ago, Bayequentist said:

Speaking of statistical learning, what do people in the stats community think of deep learning methods such as CNN, RNN, GAN, Deep Graphical Models? Thanks!

These are very nascent fields (I've heard that even computer science conferences have difficulty finding suitable reviewers for papers on theory for deep learining) and so are fruitful areas for research. But since they are so recent, deep learning is not covered in The Elements of Statistical Learning (to the best of my knowledge -- there may have been a more recent edition of it than the one I read that contains sections on deep learning).

I would also note that ESL is by no means exhaustive. The authors are frequentist statisticians, so the book only very briefly touches upon Bayesian approaches to the same problems in the book (e.g. nonparametric regression, classification, etc.). But I think as a "gentle" introduction to a variety of machine learning methods and models, the book is quite good. IMO, this book is best utilized as a basic introduction and should be supplemented by reading other tutorials and lecture notes/slides and watching video tutorials.

Edited April 21, 2019 by Stat PhD Now Postdoc

[Bayequentist]

52 minutes ago, Stat PhD Now Postdoc said:

These are very nascent fields (I've heard that even computer science conferences have difficulty finding suitable reviewers for papers on theory for deep learining) and so are fruitful areas for research. But since they are so recent, deep learning is not covered in The Elements of Statistical Learning (to the best of my knowledge -- there may have been a more recent edition of it than the one I read that contains sections on deep learning).

I would also note that ESL is by no means exhaustive. The authors are frequentist statisticians, so the book only very briefly touches upon Bayesian approaches to the same problems in the book (e.g. nonparametric regression, classification, etc.). But I think as a "gentle" introduction to a variety of machine learning methods and models, the book is quite good. IMO, this book is best utilized as a basic introduction and should be supplemented by reading other tutorials and lecture notes/slides and watching video tutorials.

The authors did publish a more recent book: Computer Age Statistical Inference, which has a better balance between frequentist and Bayesian approaches. There are also roughly 20 pages on Neural Networks and Deep Learning.

[biostat_student]

On 4/20/2019 at 10:53 AM, Stat PhD Now Postdoc said:

"The Elements of Statistical Learning" is a good book and I found it to be pretty accessible. If you have already finished the first year of your PhD program, you probably can just start reading it (and then refer to other references when you need to). I think it's a good idea to read this book from start to finish at a high-level (i.e. read it enough to understand conceptually what's going on rather than focusing on the nitty gritty technical details). Then you can refer back to parts of it later and read those sections much more carefully if they are related to your research. I never did any of the problems at the end of the chapter, but I don't think that would have helped much for research anyway.

 

Thanks for the advice @Stat PhD Now Postdoc! Were there any specific resources you used that helped you throughout your reading, or was it more just googling individual topics when you needed further explanation? Were you able to skip around based on topics of interest, or was it better to read sequentially?

[Stat Assistant Professor]

11 hours ago, biostat_student said:

Thanks for the advice @Stat PhD Now Postdoc! Were there any specific resources you used that helped you throughout your reading, or was it more just googling individual topics when you needed further explanation? Were you able to skip around based on topics of interest, or was it better to read sequentially?

I read it from start to finish, but you can probably skip around after you've read the introductory chapters on regression and classification. As for supplementary materials, I did mainly search Google and Youtube. I found watching video tutorials to be particularly helpful.