The lessons of two decades of quantitative research, applied to the field’s most seductive tool
Machine learning has produced results in image recognition and language that would have seemed like science fiction a decade ago. It is natural to assume that the same techniques, pointed at financial markets, will produce a comparable leap. In my experience, they usually produce something else: a model that looks brilliant in research and disappoints the moment it meets live capital. The tools are not the problem. The problem is that markets violate almost every assumption that makes machine learning work elsewhere.
I lead quantitative research, and I use these methods, so this is not a rejection of them. It is a warning about the specific and repeatable way they fail in finance, and a description of what I have found actually holds up.
1. The signal-to-noise problem is the whole problem
In most successful applications of machine learning, the signal is strong, and the noise is modest. A photograph of a cat contains overwhelming evidence that it is a cat. Financial returns are the opposite. The vast majority of price movement is noise, and the predictable component is faint and fleeting. When you train a flexible model on data that is mostly noise, it does what flexible models do: it fits the noise, mistakes it for signal, and reports spectacular in-sample accuracy that means nothing.
This single fact explains most machine learning failures in trading. The methods were designed for regimes where the pattern dominates the randomness. Markets are a regime where the randomness dominates the pattern, and no amount of model sophistication changes that ratio.
A model that fits noise will always look smarter than a model that respects it, right up until you trade it.
2. Non-stationarity: the ground moves under the model
Machine learning assumes, usually implicitly, that the process generating tomorrow’s data resembles the process that generated yesterday’s. In markets, that assumption is routinely false. The relationships a model learns are not laws of nature. They are behaviors of a system that adapts, and part of what it adapts to is the very strategies built on those behaviors. A pattern that was real can decay precisely because it was discovered and traded.
This is why a model validated on years of history can fail in months. It is not that the model was wrong when it was built. It is that the world it described stopped existing. Any serious use of these methods in finance has to treat the model as perishable and build the monitoring to detect when its edge has decayed.
3. The sample is smaller than it looks
Researchers point to millions of data points and assume they have ample data to train complex models. They rarely do. What matters is not the number of observations but the number of independent, informative events, and financial data has far fewer of those than its row count suggests. A decade of daily returns contains only a handful of genuine market regimes and a small number of true stress events. The rest is autocorrelated repetition of similar conditions.
Measured in effective sample size, the problem often has less data than a modest classification task in another field, while the models being fit have vastly more parameters. That imbalance is a recipe for overfitting, and no cross-validation scheme fully rescues it. The honest response is to fit simpler models and to be deeply suspicious of any result that depends on the model’s flexibility.
4. Complexity is a liability, not an asset
There is a cultural pull toward the most powerful available model, on the theory that more capacity can only help. In low signal-to-noise, non-stationary environments the opposite is true. Every additional parameter is another opportunity to fit noise, and a more complex model degrades faster when conditions shift because it has memorized more of a world that no longer exists.
The most robust strategies I have run are not the most sophisticated. They are built on a small number of economically sensible variables, with heavy regularization and a strong prior toward simplicity. Complexity is something to be spent reluctantly and only where the data genuinely earns it, which in markets is seldom.
5. Cross-validation leaks in ways nobody admits
Standard validation assumes observations are independent. Financial time series are not. Returns are autocorrelated, features overlap in time, and information from the future can seep into the training set through ordinary preprocessing. The result is validation performance that is optimistic, sometimes wildly so, and a model that looks robust on paper and is fragile in practice.
Guarding against this requires validation designed for time series: strict temporal separation, purging of overlapping samples, and embargo periods between training and testing. Even then, the safest posture is to assume the backtest is flattering and to demand that a strategy survive out-of-sample, out-of-period, and a healthy dose of skepticism before it earns capital.
6. What actually works
None of this argues for abandoning these methods. It argues for using them the way an engineer uses a powerful but temperamental tool: with respect for its failure modes. In practice that means imposing economic structure rather than letting the model find patterns unconstrained, favoring simple and regularized models over complex ones, validating with methods built for dependent data, and treating every model as something that will decay and must be monitored.
Used this way, machine learning is a genuine addition to the quantitative toolkit, particularly for problems like execution and short-horizon microstructure where the data is richer and the noise less overwhelming. Used the way it is used in other fields, pointed at raw returns with maximum flexibility and a trusting backtest, it is a reliable way to lose money with great confidence. The difference is not the algorithm. It is the discipline surrounding it.
About the author
Giulio M. Occhionero is Head of Quantitative Research and Development at IRH Global Trading in Abu Dhabi, where he leads systematic strategy deployment and quantitative research across US equities, options, and digital assets. He holds degrees in nuclear engineering and has spent more than two decades in quantitative finance across roles in Italy, the United States, and the United Arab Emirates. He writes regularly on systematic trading, market microstructure, and the engineering side of investment management.