Common backtesting mistakes and how to avoid them

A backtest can be wrong in two ways. It can compute the wrong numbers, which is rare with modern tools, or it can compute the right numbers for the wrong question, which is common and much harder to notice. Almost every backtesting mistake is a version of the second problem: information from the future leaks into decisions that were supposed to be made in the past, or the sample is chosen so that the strategy looks good, or the numbers are read to mean something they do not. This guide covers the mistakes that matter most, in rough order of how often they fool people.

1. Overfitting

What it is. Overfitting, also called curve fitting or data snooping, is tuning a strategy's parameters until the historical result looks good. Every parameter you adjust in response to the backtest makes the strategy fit the specific history more tightly and generalize less. At the limit, a rule with enough parameters can be made to look brilliant on any history, and it will have no relationship to the future.

What it looks like. A moving average lookback of 187 days. A momentum rule that excludes one specific month. A Sharpe ratio above 1.5 for a long-only equity strategy. A universe of seven ETFs with no stated reason for the seventh. A result that gets worse when you round the parameters.

Why it is dangerous. The backtest is not lying; it is accurately reporting that this exact rule did well on this exact history. The problem is that you searched for that rule by looking at the history, so the good result was guaranteed by the search, not earned by the idea. Bailey, Borwein, López de Prado, and Zhu's paper Pseudo-Mathematics and Financial Charlatanism shows how few trials it takes to produce an impressive backtest by chance.

How to avoid it. Write the rule down before you test it, with a reason for each parameter. Prefer round, conventional values: 200 days because that is what the literature uses, not 187 because it scored highest. After the first test, change each parameter a little and confirm the result degrades gracefully rather than collapsing. Count your parameters and be suspicious of anything above three or four. Most of all, limit the number of things you try; every additional variation is another lottery ticket, and one of them will win by chance.

No software can prevent overfitting. It is a discipline problem. What a tool can do is make the discipline easier by keeping the description of every variation visible, so you can see how many things you have tried.

2. Lookahead bias

What it is. Lookahead bias is using information in a decision that would not have been available at the time of the decision. The strategy sees the future, and its results are impossible to reproduce live.

What it looks like. Computing a monthly signal from the month's closing price and then trading at the month's opening price. Using a fund's total return for the year to decide whether to hold it in January. Using earnings data on the fiscal period end date rather than the date it was actually published. Using a price index that was revised after the fact.

Why it is dangerous. It is silent. The code runs, the numbers are computed correctly, and the result is often spectacular, because knowing the future is a very good strategy. Lookahead is the most common reason a live strategy dramatically underperforms its backtest.

How to avoid it. Use a backtesting engine that enforces point-in-time data access, so the code cannot read a price from a date later than the decision date. ENSEMBLE's execution harness does this: generated code sees only the history up to each rebalance date. Then check the mechanics of your own rule. If the signal uses the close, the trade should happen at that close or later, never before. If a rule seems to react suspiciously well to every turn, ask what it knew and when.

Lookahead can also enter through the description rather than the code. Choosing a universe because you know how its members performed is lookahead; so is selecting a start date after a crash you know is coming. See survivorship bias and sample selection below.

3. Survivorship bias

What it is. Survivorship bias is testing on a universe of assets that exist today, which excludes everything that failed, was delisted, or merged away. The backtest never holds the losers because the losers are not in the data.

What it looks like. A strategy that rotates among the ten largest technology stocks, tested using today's ten largest. A dividend strategy on today's dividend aristocrats. Any single-stock universe assembled in the present and tested in the past.

Why it is dangerous. The strategy inherits the returns of assets that were selected for having survived, which is a strong filter for having done well. Most of the apparent skill comes from the selection, not the rule.

How to avoid it. Use broad index funds for the universe wherever possible. An ETF that tracks the S&P 500 handles its own constituent changes point-in-time, so a strategy that holds SPY is not survivorship biased even though the index's members changed. If you must use individual securities, use a universe defined by a rule that could have been applied at the time, such as "the largest 20 stocks by market cap as of each rebalance date," and expect the results to be worse than the version that uses today's list. The strategies on this site use ETF universes for exactly this reason.

4. Ignoring costs, taxes, and slippage

What it is. Running a backtest as if trades were free and instant.

What it looks like. A strategy that trades daily and shows a large edge over buy and hold. A rotation among thinly traded funds with wide spreads. A strategy tested gross of fees that would in practice be held in a taxable account and pay short-term capital gains on every switch.

Why it is dangerous. Costs scale with turnover, and turnover is exactly what tactical strategies add. A high-frequency rule can look better than a low-frequency one gross and worse net. A backtest that ignores costs systematically favors the strategies that are most expensive to run.

How to avoid it. Use a tool that charges a realistic cost on every trade and computes metrics on the net series; ENSEMBLE's assumption is documented on the methodology page. Read the trade count and turnover on the tear sheet before reading the return. If turnover is above 200 percent a year, assume the live result will be meaningfully worse than the backtest, and if the strategy would sit in a taxable account, model the tax drag separately. A good rule of thumb: the burden of proof rises with turnover.

5. Short or regime-limited samples

What it is. Testing over a period that contains only one kind of market and drawing conclusions about the strategy in general.

What it looks like. A backtest from 2009 to 2019 showing a low maximum drawdown for an equity strategy. A trend-following strategy tested from 2009 onward that looks like a waste of money. A risk parity portfolio tested only during the forty-year decline in interest rates. A crypto strategy tested since 2020.

Why it is dangerous. Every strategy has environments it is built for and environments it is not. A sample that contains only the first kind cannot reveal the second. The 200-day moving average rule looks brilliant in a sample that includes 2008 and pointless in one that starts after it; both samples are accurate and neither is the whole story.

How to avoid it. Insist on at least two distinct regimes, including one severe bear market. For US assets, start no later than 2007. When a young fund pushes the start later, say so explicitly and treat the result accordingly; the tear sheet names the fund that set the start. Look at the strategy's behavior in each major episode separately, using the drawdown chart, rather than trusting the summary statistics. And be honest about what is missing: a twenty-year backtest ending today contains no period of sustained high inflation except 2022, and no rising-rate regime longer than two years.

6. Misreading the metrics

What it is. Reading one number as if it described the strategy.

What it looks like. Choosing the strategy with the highest CAGR. Comparing a Sharpe ratio from one period with a Sharpe ratio from another. Treating a low maximum drawdown as evidence of safety when the sample contains no bear market. Ignoring time in market when comparing a tactical strategy to buy and hold.

Why it is dangerous. Each metric compresses the whole return path into one number and discards most of the information. CAGR discards the path. Volatility discards direction. Sharpe discards the tails. Maximum drawdown discards duration. Read alone, each one can rank two strategies in the wrong order.

How to avoid it. Read them together and in a fixed order: drawdown, then CAGR against a benchmark over the same dates, then Sharpe, then turnover, then holdings over time. The metrics explained guide defines each one and explains how it misleads. Always compare against the simplest alternative measured over identical dates; for most portfolios that is a 60/40 blend.

7. Incomplete rule specification

What it is. Testing a rule that leaves decisions implicit, so the backtest tests whatever the tool guessed.

What it looks like. "Hold SPY above its 200-day average" without saying what to hold otherwise or when to check. "Rebalance regularly." "Buy the top performers."

Why it is dangerous. The tool has to fill the gaps, and different fills produce different strategies. The one you tested may not be the one you meant, and you will not find out until the live results diverge.

How to avoid it. State the universe, the weighting, and the timing completely. Then read the generated code, or at least the tear sheet's universe and trade count, to confirm the tool understood you. The how to backtest a portfolio guide has a checklist.

8. Testing until something works

What it is. The meta-mistake: running many backtests and reporting the best one. This is overfitting at the level of the research process rather than the individual strategy.

What it looks like. Trying ten universes, five lookbacks, and three rebalancing schedules, and publishing the combination that scored highest. Trying five strategies and writing up the winner as if it were the only one considered.

Why it is dangerous. With 150 trials, the best one will look excellent by chance alone, and the summary statistics do not know how many trials produced it. The Sharpe ratio of the winner is not the Sharpe ratio you should expect going forward.

How to avoid it. Keep a record of everything you tried, including the failures. Prefer strategies whose parameters you chose for a reason before testing. When you do search across variations, expect the winner to regress substantially, and treat a result that survives on conventional parameters as far more credible than one that needs specific ones.

What a tool can and cannot do

A good backtesting platform prevents the mechanical mistakes. Point-in-time data access removes most lookahead. Total-return data that a fund enters only once it has prices removes accidental survivorship and phantom history. A cost charged on every trade removes the free-trading assumption. Complete metrics on a standard tear sheet make misreading harder. ENSEMBLE does each of these, and the methodology page describes how.

What no tool can do is stop you from asking the data what it wants to hear. Overfitting, sample selection, and testing until something works are research habits, and the only defense is to write the question and the rule down before running the test, change one thing at a time, and count how many things you have tried.

The prompt below is a reasonable strategy to test with those habits in mind: a multi-asset momentum rotation with conventional parameters and a universe of broad funds. Run it, then change one parameter and see how much the result depends on it.

Backtests are hypothetical, past performance does not guarantee future results, and ENSEMBLE is research software rather than an investment adviser.