Methodology
Backtesting vs Validation: Why The Difference Matters
A clean backtest is not a working strategy. Validation is what separates research from marketing. Here is the distinction, and why most published track records ignore it.
SIGMASEVEN ResearchIn short
Backtesting measures how a strategy would have performed on historical data. Validation tests whether that performance reflects real edge or curve-fitting. Out-of-sample testing, cross-asset replication, and parameter stability analysis are the core components. A strategy that backtests well but fails validation is noise dressed as signal.
Backtesting is the easy part
Backtesting a strategy on historical data is, at this point, a solved problem. Any practitioner with a scripting language and a data source can produce a curve. The hard part is not producing it. The hard part is knowing whether it means anything.
What validation actually requires
Validation is the process of stress-testing a strategy against the conditions that most often produce false positives. At minimum, it includes out-of-sample testing on data the strategy was never exposed to during design, replication on adjacent assets, and parameter stability checks to confirm the result is not a knife-edge fit.
Each of these tests can kill the strategy. A strategy that survives all three is meaningfully more likely to hold up in live trading.
The curve-fitting trap
Most retail backtests are indistinguishable from curve-fitting. The researcher, often unintentionally, tunes parameters until the historical equity curve looks attractive. The resulting performance is a property of the data, not of the strategy.
Validation catches this. It asks: does the strategy perform on data the designer did not touch?
Why published track records often skip this
Because validation almost always makes the numbers look worse. A backtest optimised on ten years of one asset can display eye-catching metrics. The same rules, run on a second asset without re-optimisation, usually display humbler metrics. Marketing prefers the former.
A serious quantitative provider publishes the humbler numbers and treats them as the honest expectation. Every strategy in our product suite is validated across multiple assets and out-of-sample periods before publication.
Frequently asked questions
- What is backtesting?
- Backtesting is the process of applying a trading or investing strategy to historical data to measure how it would have performed in the past.
- What is validation in quantitative finance?
- Validation is the process of confirming that backtested performance reflects a real, generalisable edge rather than overfitting to a specific historical sample.
- What is curve-fitting?
- Curve-fitting is the act of tuning a strategy's parameters until it matches historical data extremely well, usually at the cost of its ability to perform on future or unseen data.
- Why do many strategies fail after going live?
- Because they were optimised heavily on in-sample data and never properly validated. Live performance exposes the gap between curve-fitting and real edge.