Why Sample Size Matters in Backtesting: How Many Trades Do You Need?

A trader showed me a strategy that returned 340% in backtesting. Twelve trades. All winners. Profit factor infinity. Sharpe ratio through the roof. Was this a good strategy? I have absolutely no idea — and neither does anyone else. Twelve trades tells you nothing. It's the statistical equivalent of flipping a coin three times, getting heads each time, and concluding that coins always land on heads.

Why Sample Size Is Non-Negotiable

Every backtest metric is an estimate of the strategy's true performance. The true performance is what the strategy would achieve over an infinite number of trades. Your backtest gives you a sample — a finite subset of all possible trades. The accuracy of your estimate depends entirely on sample size.

The statistical relationship is clear: estimation error decreases proportionally to the square root of the sample size. To halve your estimation error, you need 4x the trades. To reduce it to a tenth, you need 100x the trades.

If your backtest shows a 55% win rate over 50 trades, the 95% confidence interval is roughly 41-69%. The true win rate could be anywhere in that range. At 41%, the strategy is a loser (depending on R:R). At 69%, it's excellent. You simply don't know which one you have.

At 500 trades, the same 55% win rate has a confidence interval of 51-59%. Now you can be reasonably confident the strategy actually wins more than it loses.

The Parameters Problem

The required sample size increases with the number of parameters in your strategy. Each parameter needs to be justified by sufficient data. A rough guideline: you need 50-100 trades per free parameter.

• Strategy with 2 parameters: minimum 100-200 trades
• Strategy with 4 parameters: minimum 200-400 trades
• Strategy with 8 parameters: minimum 400-800 trades

This explains why complex strategies with many parameters almost always overfit — the amount of data needed to properly validate them exceeds what most traders have available.

How to Increase Your Sample Size

If your strategy doesn't generate enough trades for statistical significance, you have several options:

1. Extend the data period. More years = more trades. This is the best solution when data is available. For crypto, data going back to 2017-2018 is widely available. For forex, you can get decades of data. For US stocks, minute-level data often goes back 5-10 years.

2. Test across multiple instruments. If your BTC strategy produces 80 trades over 3 years, also test it on ETH, SOL, and BNB. If each produces 60-80 trades, your combined sample is 240-320 trades. This also tests robustness across instruments — a double benefit.

3. Lower the timeframe. A strategy on daily candles might produce 50 trades per year. The same logic on 4-hour candles might produce 200. The strategy needs to actually work on the lower timeframe, but it's worth testing if your sample size is insufficient on higher timeframes.

4. Don't add unnecessary filters. Every filter you add reduces trade count. Before adding a "volume confirmation" or "session filter," ask: does this filter improve the strategy enough to justify the sample size reduction? If removing a filter increases trades from 80 to 200 while only reducing win rate from 55% to 52%, the filter probably isn't worth it — you gain much more from statistical reliability than you lose from a 3% win rate drop.

The Practical Minimum: 100 Trades

For retail traders without access to decades of institutional-quality data, here's a practical framework:

When interpreting your backtest results, always check the trade count first. If it's below 100, treat every other metric with extreme skepticism — even if the numbers look incredible.

"The plural of anecdote is not data. And twelve trades is barely an anecdote." — A statistics professor who saved me from deploying an untested strategy early in my career