Machine Learning within the scientific workflow
As I transition from theoretical physics further into machine learning research, I have increasingly found myself interested in a somewhat different level of research problem. Rather than asking only which model performs best on a particular scientific dataset, I have naturally gravitated toward the methodological question of how we should learn, infer, simulate, represent, and make decisions about complex scientific systems. Machine learning for science is often described in terms of particular applications: predicting molecular properties, solving differential equations, analysing biological data, or accelerating simulation. But, perhaps because I don't come from a purely comp-sci background, experience working in research has made me intensely aware that in many scientific settings data are limited, experiments are expensive, uncertainty matters, and mechanistic structure is available.
Maybe one way of formalising this is to imagine a scientific system with unknown parameters or latent structure \( \theta \), a simulator or mechanistic model \(S\), an experimental intervention \(\alpha\), and observations \(x\)
\[ \theta \sim p(\theta), \qquad z=S(\theta,a), \qquad x\sim p(x\mid z). \]
Since modern machine learning is rapidly developing, I find it useful to think about how different areas appear as different questions about the same system. For instance, representation learning might ask which variables \( r_\phi(x) \) capture scientifically useful structure. Inverse problems and simulation-based inference might ask how to infer \( p(\theta\mid x) \). Generative modelling may ask how to represent and sample from complex conditional distributions, often with the aim of design rather than prediction. Active learning, Bayesian optimisation, and reinforcement learning ask how the next intervention \(\alpha \) should be chosen. These descriptions are necessarily schematic, but, for the sake of discussion, they begin to show how scientific machine learning asks how known physical or mechanistic structure can be incorporated into these models rather than relearned entirely from data. Once uncertainty is treated as the central part of inference and decision-making, it is natural to ask whether uncertainty arising from computation itself should also be represented explicitly. This is the perspective taken when viewing ML through the lens of probabilistic numerical methods, which treat numerical approximation not merely as an implementation detail, but as another source of uncertainty that can in principle be quantified.
A related perspective, I have learned, comes from approximation theory. Recent work on deep versus shallow neural networks increasingly appears to give emphasis to not universal approximation itself, but the ability of deep architectures to exploit compositional structure in high-dimensional functions. This is a fascinating topic itself, and I have already started drafting a post about it. As it connects with probabilistic numerical thinking: it seems to me that, in both cases, the central object is not simply a trained predictor, but a finite approximation to an underlying mathematical object. Maybe that statement is obvious. If we frame approximation theory as asking which architectures exploit its structure efficiently, probabilistic numerics thrives in the context of asking how uncertainty arising from approximation itself should be represented.
Of course, I don't have the perspective of an ML researcher who has worked actively in the field since the 1990s and observed its many revolutions, or the connections between them that became apparent only in retrospect. But, coming from string theory, which has itself undergone a couple of especially notable revolutions since the 1980s, I do think one can draw a few useful analogies. What is striking in 2026 is how often these different areas of machine learning are increasingly converging. Scientific machine learning appears to be moving beyond supervised prediction toward generative design, learned representations, simulation-based inference, and closed-loop experimentation. In biology, for example, generative models are now being developed for structured molecular and protein design, including experimentally validated de novo antibody design, while active-learning and Bayesian experimental-design methods are being used to decide which costly experiments should be performed next. At the same time, physics-informed and scientific foundation models continue to ask how prior scientific knowledge can reduce data requirements and improve generalisation, or even how laws of physics may be encoded into increasingly general foundational models. The increasingly important object is not simply a predictive model, perhaps as in modern applications in quantitative finance, but a learning-and-decision system embedded within a scientific workflow.
I have found it useful, at least provisionally, to think of this territory as a kind of applied machine learning theory: theoretical and mathematical questions that arise from the constraints of real scientific problems. The emphasis is not on theory detached from applications, nor purely on mathematics, but a constant combination and collaboration between both, and on applying established architectures mechanically, so as to understand which representations, probabilistic assumptions, inference procedures, computational approximations, and decision rules are appropriate for a given scientific problem. This seems to be a particularly fertile interface between machine learning, statistics, numerical analysis, scientific computing, and the mathematical sciences. I have to say, I absolutely love it.