We may earn a commission through these partner links. Demo funds are virtual; access and country eligibility depend on the platform terms. Disclosure
A strategy can look excellent because it was selected to fit past noise. The number of choices tried matters when interpreting the best historical result.
The selection problem
If many rule variants are compared on the same data, one can appear strong by chance. Selecting that winner and reporting its result without the search history overstates the independence of the evidence.
Research by Bailey and co-authors develops methods for assessing backtest overfitting. This article explains the selection issue; it does not claim to have applied their full statistical framework to a particular strategy.
Count more than the visible parameters
Changing indicators, lengths, sessions, instruments, dates, payout assumptions and exclusion rules all creates choices. Manual trial and error can search a large space even if the final rule looks simple.
Keep a candidate log. A report showing one final configuration is incomplete if dozens of discarded versions influenced its selection.
An original thought experiment
Imagine one hundred fair independent signal generators, each producing five binary observations with equal chances of success and failure. A particular generator has probability 1/32 of five successes.
The expected number of perfect five-observation records is 100/32 = 3.125. Seeing a few perfect short records among many candidates would therefore not, by itself, establish predictive ability. This is a simplified probability example, not a model of actual markets.
A development record
| Record | Purpose |
|---|---|
| Candidate rule | Shows what was tried |
| Data segment | Separates design and evaluation |
| Selection criterion | Explains why the winner was chosen |
| Rejected variants | Preserves the search history |
| Final evaluation | Tests the frozen selected rule |
A holdout helps but is not magic
Keeping evaluation data separate is useful, but repeatedly checking it and redesigning the rule turns it into development data. Dependence, overlapping outcomes and non-stationarity also complicate inference.
Use a method appropriate to the dataset and disclose its limitations. Do not present one favourable holdout as an unconditional certificate of robustness.
Publish uncertainty rather than a polished winner alone
Report the candidate count, selection process, net payoff assumptions and independent evaluation. Include failed tests and missing data.
A historical chart can investigate an idea. It cannot establish guaranteed live returns, payment reliability or the accuracy of future signals. More optimisation should increase the burden of evidence, not the confidence of the marketing language.
The chance of finding at least one perfect short record
In the simplified independent-generator experiment above, the probability that none of 100 candidates wins all five observations is (31/32)100. Therefore the probability that at least one looks perfect is 1 − (31/32)100, approximately 95.82%. That is different from the expected count of 3.125 perfect records.
This calculation assumes independent generators and independent fair observations. Actual strategies often share data and rules, so it is not a numerical estimate of real backtest-overfitting probability. Its purpose is to show how searching many candidates changes the meaning of a selected impressive result.
Make the hidden search visible
A candidate log should record manual changes as well as automated optimisation. Testing four indicator lengths across three sessions and five instruments creates 60 combinations before adding entry filters, date ranges or payout assumptions. The combinations may be correlated, but concealing them still makes the final winner appear less selected than it was.
| Design decision | Record before selection | Risk if hidden |
|---|---|---|
| Indicator setting | All tried lengths and thresholds | Winner looks like the only idea tested |
| Trading hours | Sessions included and excluded | Weak periods disappear |
| Instrument universe | Eligible assets and rejected cases | Only favourable markets remain |
| Evaluation endpoint | Fixed dates and stopping rule | Testing ends at a lucky peak |
| Economic model | Stake, payout, costs and settlement | Signal accuracy substitutes for net results |
Use a chronological development protocol
- State the hypothesis and candidate space before inspecting the final evaluation period.
- Use development data to choose rules and record every material revision.
- Keep selection criteria fixed rather than switching from return to accuracy when convenient.
- Freeze the chosen implementation and its economic assumptions.
- Evaluate on untouched data and publish the result, including a failure.
If the final evaluation inspires another redesign, label that period as used development information. A later genuine evaluation needs data that has not already guided the revision. Repeatedly naming the same dates “out of sample” does not restore their independence. Walk-forward testing also needs a predeclared schedule and selection procedure.
Check the execution model as well as the parameters
TradingView’s strategy documentation distinguishes overfitting, selection bias and lookahead bias. They can coexist but are different problems. A rule tuned to noise is an overfitting concern; a rule using information unavailable at entry has an information-timing problem even if no parameters were optimised.
For fixed-outcome contracts, directional accuracy alone is insufficient. The backtest needs the actual intended entry convention, expiry, settlement comparison, payout, tie treatment and unavailable-entry handling. A generic price-based strategy report does not automatically model those economics. State which assumptions were observed and which were imposed.
Robustness checks can become further optimisation
Nearby parameter checks and tests across periods can reveal sensitivity. However, if you inspect those results and select the most flattering region, the checks have also influenced design. Record that decision. A broad plateau of acceptable historical results may be more informative than one isolated peak, but it is not a certificate of future profitability.
Use the research methods hub and calculation tools to document assumptions and reproduce arithmetic. Keep a version identifier for code, data and the rulebook so a reviewer can tell which configuration produced each report.
Backtest overfitting FAQs
Is a simple strategy immune to overfitting?
No. A simple final rule may be the winner of a large hidden search. Its selection history still matters.
Does a positive holdout prove the strategy works?
No. Sample uncertainty, data dependence, changing conditions and execution assumptions remain. Report what the holdout actually tested.
Should failed variants be deleted?
Keep them in the development record. They explain the search process and prevent the final report from presenting a selected winner as an untouched original hypothesis.
Sources and research scope
- Bailey, Borwein, López de Prado and Zhu: The Probability of Backtest Overfitting
- TradingView: strategy testing, selection bias and overfitting
Public-source review: 25 October 2026. Examples are illustrative. Documentary research does not represent funded-account testing. Rules must be checked against the current product, entity and customer location.
Related research
Research methods · Research methodology · Calculation tools
Risk: Fixed-outcome contracts can lose the entire stake. No chart pattern, calculator or demo result guarantees future returns.