This article is our own summary and commentary on published research. The original work is credited in full under "Sources & credits" below. Nothing here is investment advice.
A backtest is a story about the past. The trouble is that it is very easy to tell a flattering one. Change a moving-average length, add a filter, shift the stop-loss, and run it again. After enough tries, something will show a smooth equity curve and an impressive Sharpe ratio. Most of the time, that result says more about the search than about the market.
What the research found
In 2014, mathematicians David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu published a paper in the Notices of the American Mathematical Society on exactly this problem. They called it backtest overfitting.
Their central point is simple. If you test many configurations of a strategy on the same historical data and keep the best one, a high in-sample score is easy to reach even when the strategy has no real edge. The more configurations you try, the more likely it is that the winner is overfit. And because people rarely report how many variations they tested, an investor looking at the final backtest has no way to judge how much luck is baked into it.
The authors also propose a minimum backtest length: the shorter the history relative to the number of trials, the less a good result means. They are careful to say this is a necessary condition, not a sufficient one. A long backtest does not prove a strategy works; a short one with many trials is a warning sign.
Why this matters more in options
Index options make the problem worse. Weekly expiries, many strikes and many parameters (entry time, strike distance, stop rules, hedge rules) multiply the number of variations a trader can try. Each extra knob is another chance to fit noise. Costs also bite harder: slippage on a cheap option can be a large share of its price.
The checks we run
None of this means backtests are useless. It means they need to be treated as evidence that can be challenged. Before any strategy reaches our own capital, it has to pass these tests:
- Count the trials. We record every variation tested, not just the winner. A result found after 500 tries is judged far more harshly than one found after five.
- Keep data the model has never seen. Out-of-sample periods, and walk-forward testing where the strategy is re-fitted and then run on the next unseen window.
- Stress the parameters. If performance collapses when a parameter moves slightly, the edge is probably an artefact. Robust strategies have broad, flat regions of good results.
- Charge real costs. Brokerage, taxes, slippage, fill probability and margin, including expiry-day behaviour.
- Ask why it should work. A pattern with no economic reason behind it gets extra scrutiny, however good its numbers look.
Most ideas do not survive this. That is the point: the cheapest place to lose money is in a backtest.
Sources & credits
- David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", Notices of the American Mathematical Society, Vol. 61, No. 5 (May 2014), pp. 458–471. DOI 10.1090/noti1105. Preprint (PDF)
The research findings belong to their authors. This post summarises them in our own words and adds our own commentary.