Two essential texts for applied machine learning theory
I often see the question: which textbooks should I study to learn machine learning properly?
There is now no shortage of introductory material. Machine learning courses, lecture notes, online tutorials, and practically oriented books have proliferated over the past decade, and many of them provide excellent introductions to particular algorithms or software frameworks. However, if you're like me and you're not interested in only understanding how to apply models, but to understand machine learning theory, and properly come to grips with machine learning at a mathematical level, there are two books I continue to regard as essential:
- Trevor Hastie, Robert Tibshirani and Jerome Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. [Freely available from the authors' website].
- Christopher M. Bishop, Pattern Recognition and Machine Learning. [Freely available from Microsoft Research].
I like the idea of applied machine learning theory: understanding the mathematical structure of learning algorithms while remaining concerned with the modelling problems for which those algorithms are actually constructed.
There is a difference between simply wanting to understand how to apply ML frameworks, and in the desire to understand why models work, where they break, and how to develop them in the context of a complex problem. I suppose, coming from a theoretical physics PhD, I am interested in machine learning not merely as a collection of predictive technologies, or in terms of a black box that spits out cool results; but as a computational tool and also part of a wider quantitative framework.
In computational physics and biology, for example, we are often trying to infer structure from incomplete, noisy, and finite observations of complicated systems. Increasingly, we need more advanced quantitative methods and modelling techniques, and the march of AI/ML for scientific discovery will prove invaluable. At the same time, progress inevitably always leads back to mathematics. And, really, machine learning is mathematics. We need probability to describe uncertainty. Statistics to reason about finite data. Linear algebra to describe high-dimensional representations. Optimisation to fit models. And increasingly, ideas from numerical analysis, information theory, geometry and dynamical systems enter as well. The idea of applied machine learning theory situates itself amongst all of this, and I think that's why I love it.
Neither ESL nor PRML contains the final word on modern machine learning, nor could any textbook. But I would say that both of these are substantial books that contribute to a good foundation. The Elements of Statistical Learning develops the subject primarily from the viewpoint of statistical framing, estimation, and prediction. Pattern Recognition and Machine Learning places probability theory and probabilistic modelling much closer to the centre of the subject. Together they provide a vocabulary with which to approach new methods.
When I have some more time, I would like to review both of them respectively.