Definition
Overfitting
Overfitting is tuning a strategy to historical noise rather than signal. How to avoid overfitting trading strategies: the warning signs, and why it is the default outcome of optimisation.
Overfitting is the result of tuning a strategy until it describes the noise in a historical sample rather than any repeatable structure in the market. The tell is that performance collapses on data the parameters were not chosen from. It is not a rare failure mode but the default outcome of unconstrained optimisation: given enough parameters and enough attempts, any data set can be fitted perfectly.
- Also known as
- curve fitting · data snooping
How it is calculated
Symptoms of an overfitted strategy:
· Performance collapses on data the parameters were not chosen from
· Small parameter changes produce large performance changes
· More rules and more conditions than the sample can support
· An equity curve that is smooth in-sample and jagged out-of-sample
Every backtest you run costs something
Testing a hundred variants and keeping the best one guarantees a good in-sample result even if none of the variants has an edge — the winner is selected on the noise. This is why the number of configurations tried has to be counted and discounted for, and why a strategy that survived one honest test is stronger evidence than one that won a thousand.
Robustness looks boring
A strategy whose performance degrades smoothly as parameters move away from optimum is more likely to be describing something real than one with a sharp peak. If a five-period lookback is profitable and a six-period lookback is not, the strategy has found an artefact of the sample rather than a property of the market.
How it gets misread
Out-of-sample testing is treated as a cure. It is only a test, and it stops working the moment its results are used to revise the strategy: once you have looked at the holdout and changed something, the holdout has become in-sample and the honest evidence is gone.
Sources
- Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance — Notices of the American Mathematical Society (Bailey, Borwein, López de Prado, Zhu)
- … and the Cross-Section of Expected Returns — Campbell R. Harvey, Yan Liu, Heqing Zhu — Duke University (published in the Review of Financial Studies)
- Data-Snooping Biases in Tests of Financial Asset Pricing Models — National Bureau of Economic Research (Andrew W. Lo, A. Craig MacKinlay)
Definitions are educational. Nothing here is investment advice, and no metric described on this page predicts future results.
Definitions are the easy part
Knowing what drawdown means is not the same as having a system that halts on it. QANTERION applies these limits while a strategy runs.