A backtest can be wrong in two ways. It can compute the wrong numbers, or it can compute the right numbers for a misleading question. Almost every backtesting mistake is a version of the second problem: information from the future leaks into decisions that were supposed to be made in the past, or the sample is chosen so that the strategy looks good, or the numbers are read to mean something they do not. This guide covers several important mistakes and ways to investigate them.
1. Overfitting
What it is. Overfitting, also called curve fitting or data snooping, is tuning a strategy's parameters until the historical result looks good. Every parameter you adjust in response to the backtest makes the strategy fit the specific history more tightly and generalize less. At the limit, a rule with enough parameters can be made to look brilliant on any history, and it will have no relationship to the future.
What it looks like. A moving average lookback of 187 days. A momentum rule that excludes one specific month. A Sharpe ratio above 1.5 for a long-only equity strategy. A universe of seven ETFs with no stated reason for the seventh. A result that gets worse when you round the parameters.
Why it is dangerous. The backtest is not lying; it is accurately reporting that this exact rule did well on this exact history. The problem is that you searched for that rule by looking at the history, so the good result was guaranteed by the search, not earned by the idea. Bailey, Borwein, López de Prado, and Zhu's paper Pseudo-Mathematics and Financial Charlatanism shows how few trials it takes to produce an impressive backtest by chance.
How to avoid it. Write the rule down before you test it, with a reason for each parameter. Prefer round, conventional values: 200 days because that is what the literature uses, not 187 because it scored highest. After the first test, change each parameter a little and confirm the result degrades gracefully rather than collapsing. Count your parameters and be suspicious of anything above three or four. Most of all, limit the number of things you try; every additional variation is another lottery ticket, and one of them will win by chance.
No software can prevent overfitting. It is a discipline problem. What a tool can do is make the discipline easier by keeping the description of every variation visible, so you can see how many things you have tried.
2. Lookahead bias
What it is. Lookahead bias is using information in a decision that would not have been available at the time of the decision. The strategy sees the future, and its results are impossible to reproduce live.
What it looks like. Computing a monthly signal from the month's closing price and then trading at the month's opening price. Using a fund's total return for the year to decide whether to hold it in January. Using earnings data on the fiscal period end date rather than the date it was actually published. Using a price index that was revised after the fact.
Why it is dangerous. It is silent. The code runs, the numbers are computed correctly, and the result is often spectacular, because knowing the future is a very good strategy. Lookahead is the most common reason a live strategy dramatically underperforms its backtest.
How to avoid it. Inspect when a signal becomes available and when the model executes it. ENSEMBLE runs user-defined or generated code; successful execution does not certify that the strategy avoids lookahead. If a signal uses a closing price, review whether the assumed trade could actually have happened at the modeled price. Check both the implementation and the data assumptions in the methodology.
Lookahead can also enter through the description rather than the code. Choosing a universe because you know how its members performed is lookahead; so is selecting a start date after a crash you know is coming. See survivorship bias and sample selection below.
3. Survivorship bias
What it is. Survivorship bias is testing on a universe of assets that exist today, which excludes everything that failed, was delisted, or merged away. The backtest never holds the losers because the losers are not in the data.
What it looks like. A strategy that rotates among the ten largest technology stocks, tested using today's ten largest. A dividend strategy on today's dividend aristocrats. Any single-stock universe assembled in the present and tested in the past.
Why it is dangerous. The strategy inherits the returns of assets that were selected for having survived, which is a strong filter for having done well. Most of the apparent skill comes from the selection, not the rule.
How to avoid it. Distinguish investing in a fund over its observed history from reconstructing its historical constituents. A current list of securities is not a historical investable universe. ENSEMBLE does not promise exhaustive delisted-asset or point-in-time constituent coverage. Inspect available dates and missing data, and label any synthetic extension explicitly.
4. Ignoring costs, taxes, and slippage
What it is. Running a backtest as if trades were free and instant.
What it looks like. A strategy that trades daily and shows a large edge over buy and hold. A rotation among thinly traded funds with wide spreads. A strategy tested gross of fees that would in practice be held in a taxable account and pay short-term capital gains on every switch.
Why it is dangerous. Costs scale with turnover, and turnover is exactly what tactical strategies add. A high-frequency rule can look better than a low-frequency one gross and worse net. A backtest that ignores costs systematically favors the strategies that are most expensive to run.
How to avoid it. Inspect what the model actually measures. ENSEMBLE does not automatically include every trading cost, tax or market impact. A requested fee overlay is distinct from modeling trade execution costs. Read the script, selected result view and turnover, then record which costs remain unmodeled; see the methodology.
5. Short or regime-limited samples
What it is. Testing over a period that contains only one kind of market and drawing conclusions about the strategy in general.
What it looks like. A backtest from 2009 to 2019 showing a low maximum drawdown for an equity strategy. A trend-following strategy tested from 2009 onward that looks like a waste of money. A risk parity portfolio tested only during the forty-year decline in interest rates. A crypto strategy tested since 2020.
Why it is dangerous. Every strategy has environments it is built for and environments it is not. A sample that contains only the first kind cannot reveal the second. The 200-day moving average rule looks brilliant in a sample that includes 2008 and pointless in one that starts after it; both samples are accurate and neither is the whole story.
How to avoid it. Examine multiple market conditions where history supports them. State which assets, dates and assumptions constrain the sample. A synthetic backfill can extend a series but does not turn that extension into observed fund performance. Compare subperiods using the same inputs and explain what remains untested.
6. Misreading the metrics
What it is. Reading one number as if it described the strategy.
What it looks like. Choosing the strategy with the highest CAGR. Comparing a Sharpe ratio from one period with a Sharpe ratio from another. Treating a low maximum drawdown as evidence of safety when the sample contains no bear market. Ignoring time in market when comparing a tactical strategy to buy and hold.
Why it is dangerous. Each metric compresses the whole return path into one number and discards most of the information. CAGR discards the path. Volatility discards direction. Sharpe discards the tails. Maximum drawdown discards duration. Read alone, each one can rank two strategies in the wrong order.
How to avoid it. Read them together and in a fixed order: drawdown, then CAGR against a benchmark over the same dates, then Sharpe, then turnover, then allocation over time. The metrics explained guide defines each one and explains how it misleads. Always compare against the simplest alternative measured over identical dates; for most portfolios that is a 60/40 blend.
7. Incomplete rule specification
What it is. Testing a rule that leaves decisions implicit, so the backtest tests whatever the tool guessed.
What it looks like. "Hold SPY above its 200-day average" without saying what to hold otherwise or when to check. "Rebalance regularly." "Buy the top performers."
Why it is dangerous. The tool has to fill the gaps, and different fills produce different strategies. The one you tested may not be the one you meant, and you will not find out until the live results diverge.
How to avoid it. State the universe, the weighting, and the timing completely. Then read the generated code, or at least the tear sheet's universe and trade count, to confirm the tool understood you. The how to backtest a portfolio guide has a checklist.
8. Testing until something works
What it is. The meta-mistake: running many backtests and reporting the best one. This is overfitting at the level of the research process rather than the individual strategy.
What it looks like. Trying ten universes, five lookbacks, and three rebalancing schedules, and publishing the combination that scored highest. Trying five strategies and writing up the winner as if it were the only one considered.
Why it is dangerous. With 150 trials, the best one will look excellent by chance alone, and the summary statistics do not know how many trials produced it. The Sharpe ratio of the winner is not the Sharpe ratio you should expect going forward.
How to avoid it. Keep a record of everything you tried, including the failures. Prefer strategies whose parameters you chose for a reason before testing. When you do search across variations, expect the winner to regress substantially, and treat a result that survives on conventional parameters as far more credible than one that needs specific ones.
What a tool can and cannot do
ENSEMBLE preserves model code, inputs and runs so you can inspect how a result was produced. That research record helps investigate mistakes; it does not certify the correctness of every strategy or remove data and execution limitations. The methodology describes the boundaries.
What no tool can do is stop you from asking the data what it wants to hear. Overfitting, sample selection, and testing until something works are research habits, and the only defense is to write the question and the rule down before running the test, change one thing at a time, and count how many things you have tried.
The prompt below is a reasonable strategy to test with those habits in mind: a multi-asset momentum rotation with conventional parameters and a universe of broad funds. Run it, then change one parameter and see how much the result depends on it.
Backtests are hypothetical, past performance does not guarantee future results, and ENSEMBLE is research software rather than an investment adviser.
Frequently asked questions
- What is the most common backtesting mistake?
- Overfitting is a major risk that software alone cannot prevent. Every parameter you tune to improve a backtest makes the result describe the past more closely and the future less.
- How can I tell if a backtest is overfit?
- Change the parameters slightly. A robust strategy degrades gracefully when the lookback moves from 200 to 180 days or the rebalancing moves from monthly to quarterly. An overfit one falls apart. Record how many variations you tried. Parameter count alone does not establish whether a strategy is overfit.
- Does ENSEMBLE prevent lookahead bias?
- Do not assume a successful run is free of lookahead. Inspect the script, signal dates, execution timing and inputs. Selecting assets because you already know how they performed can introduce bias even when the code uses prices correctly.
- How long should a backtest be?
- Use the relevant history actually available, examine different market conditions, and state what the sample misses. Longer history is not automatically better evidence, particularly when it relies on synthetic backfills or a changing strategy.
Related
- BacktestingBacktest metrics explained: CAGR, volatility, Sharpe ratio, and max drawdown
Read ENSEMBLE backtest metrics with their actual conventions: CAGR, annualized volatility, a zero-risk-free Sharpe calculation, maximum drawdown and trading activity. Compare matched windows and inspect assumptions.
- BacktestingHow to backtest a portfolio without writing code
Research two assets, build an allocation model, inspect its saved results, and compare one change. A practical first workflow for DIY investors using ENSEMBLE.
- StrategiesThe 200-day moving average strategy: rules, evidence, and a backtest
Define a moving-average allocation rule, specify signal and execution timing, and inspect its tradeoffs in a model run. An illustrative prompt helps you build and compare your own version.
- StrategiesDual momentum: the rules, the evidence, and a backtest
Define relative and absolute momentum separately, inspect a public model, and compare a parameter change. The example explains the research workflow without promising a historical or future result.